Image processing device, image processing method, and program

Abstract

To provide an image processing device capable of determining an imaging position candidate suitable for measuring a three-dimensional shape of a region where occlusion occurs.SOLUTION: An image processing device includes a viewpoint conversion unit, a calculation unit, and a determination unit. The viewpoint conversion unit converts a real depth map obtained from an image captured by a three-dimension camera attached to a robot arm into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints. The calculation unit calculates an area of a missing region in which distance information is missing for each of the plurality of virtual depth maps. The determination unit determines one or more virtual viewpoints corresponding to one or more virtual depth maps in which an area of the missing region is relatively large as one or more imaging position candidates of the three-dimension camera.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known an image processing device that measures the three-dimensional shapes of a plurality of workpieces based on images obtained from a three-dimensional camera, and controls the workpiece picking operation by a robot based on the measurement results.

[0003] When viewed from a certain viewpoint, occlusion may occur, where a part of a workpiece is hidden by another object. Therefore, a method of moving the imaging position of a three-dimensional camera has been proposed so that the location where occlusion occurs can be measured. It is desirable to appropriately determine the imaging position of the three-dimensional camera according to the current state of the workpiece so that delays in takt or leaving the workpiece behind are suppressed.

[0004] For example, JP 2019-207140 A (Patent Document 1) discloses a technology for extracting a missing area where height information is missing from a measured three-dimensional shape, and determining the imaging position of a three-dimensional camera based on the height information around the missing area so as to make it easier to obtain height information of the missing area. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-207140 A Summary of the Invention [Problem to be solved by the invention]

[0006] The technology disclosed in Patent Document 1 focuses only on the highest defective area when there are multiple defective areas in the measured three-dimensional shape, and determines the imaging position so as to easily obtain height information of the defective area of ​​interest. Therefore, for areas where occlusion occurs other than the defective area of ​​interest, it may not be possible to measure the three-dimensional shape from the image corresponding to the next imaging position.

[0007] The present disclosure has been made in consideration of the above-mentioned situation, and its purpose is to provide an image processing device, an image processing method, and a program capable of determining candidate imaging positions suitable for measuring the three-dimensional shape of an area where occlusion is occurring. [Means for solving the problem]

[0008] According to an example of the present disclosure, an image processing device includes a viewpoint conversion unit, a calculation unit, and a determination unit. The viewpoint conversion unit converts a real depth map obtained from an image captured by a three-dimensional camera attached to a robot arm into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints. The calculation unit calculates an area of ​​a missing region where distance information is missing for each of the plurality of virtual depth maps. The determination unit determines one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area among the plurality of virtual depth maps as one or more next imaging position candidates for the three-dimensional camera.

[0009] According to this disclosure, one or more imaging position candidates are determined from among a plurality of virtual viewpoints, taking into consideration the area of ​​a missing area in each virtual depth map. The missing area in the virtual depth map corresponds to an area where occlusion occurs when viewed from an imaging position corresponding to a captured image used to generate the real depth map. This allows the image processing device to determine imaging position candidates suitable for measuring the three-dimensional shape of the area where occlusion occurs.

[0010] In the above disclosure, the image processing device further includes a range setting unit that sets a movable range of the three-dimensional camera in response to an input to the user interface. The viewpoint conversion unit arranges a plurality of virtual viewpoints within the movable range.

[0011] According to this disclosure, the user can set the movable range so as to exclude positions that are obviously inappropriate for shooting with a three-dimensional camera, for example. This prevents a position that is obviously inappropriate for shooting with a three-dimensional camera from being selected as a virtual viewpoint. If an obviously inappropriate position is selected as a virtual viewpoint, the generation of a virtual depth map corresponding to the virtual viewpoint and the processing of the virtual depth map corresponding to the virtual viewpoint are unnecessary. Therefore, by setting the movable range, the time required for such unnecessary processing is reduced.

[0012] In the above disclosure, the missing region does not include the region outside the virtual frame line corresponding to the image frame of the real depth map in each of the multiple virtual depth maps.

[0013] According to this disclosure, the image processing device can avoid identifying an area that is not caused by occlusion as a missing area.

[0014] In the above disclosure, the calculation unit determines, as a missing region, a region in which a predetermined number or more of pixels with missing distance information are connected.

[0015] According to this disclosure, the image processing device can avoid identifying a minute area in which distance information is missing due to the influence of noise or the like as a missing area.

[0016] In the above disclosure, the image processing device further includes a priority setting unit and an output unit. The priority setting unit sets a priority for each of the one or more imaging position candidates based on one or more indexes including at least area. The output unit outputs an inquiry as to whether or not the three-dimensional camera can be moved to an imaging position candidate selected from the one or more imaging position candidates to a robot controller that controls the robot arm. The output unit selects the imaging position candidate to be the subject of the inquiry from the one or more imaging position candidates in descending order of priority.

[0017] According to this disclosure, the image processing device can check whether or not the three-dimensional camera can move to the imaging position candidates in descending order of priority.

[0018] In the above disclosure, the one or more indicators include a movement amount of a robot arm for moving a three-dimensional camera to a target imaging position candidate among one or more imaging position candidates.

[0019] According to this disclosure, the image processing device can set priorities taking into account the amount of movement of the robot arm.

[0020] According to another example of the present disclosure, an image processing method includes one or more processors converting a real depth map obtained from an image captured by a three-dimensional camera attached to a robot arm into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints, the one or more processors calculating an area of ​​a missing region where distance information is missing for each of the plurality of virtual depth maps, and the one or more processors determining, as one or more next candidate imaging positions for the three-dimensional camera, one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area among the plurality of virtual depth maps.

[0021] According to yet another example of the present disclosure, a program causes a computer to execute the image processing method described above. According to these disclosures, the image processing method or program can also determine imaging position candidates suitable for measuring the three-dimensional shape of an area where occlusion occurs. Effect of the Invention

[0022] According to the present disclosure, an image processing device, an image processing method, and a program can determine imaging position candidates suitable for measuring the three-dimensional shape of an area where occlusion occurs. [Brief description of the drawings]

[0023] [Figure 1] 1 is a schematic diagram showing an overall configuration of a system according to an embodiment; [Diagram 2] 2 is a schematic diagram illustrating an example of a hardware configuration of the image processing device illustrated in FIG. 1. [Diagram 3] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a robot controller according to the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing device and a robot controller. [Diagram 5] FIG. 11 is a diagram illustrating a method for generating a model image. [Figure 6] 10 is a diagram illustrating an example of a recognition result stored in a recognition result storage unit. FIG. [Figure 7] FIG. 13 is a diagram illustrating an example of a model image set. [Figure 8] 2 is a flowchart showing an example of a process flow of the system shown in FIG. 1. [Figure 9] 9 is a flowchart showing a process flow of a first embodiment of a subroutine of step S8 shown in FIG. 8. [Figure 10] FIG. 2 is a diagram illustrating an example of a plurality of virtual viewpoints. [Figure 11] 11 is a diagram showing an example of a virtual depth map seen from the virtual viewpoint shown in FIG. 10. FIG. [Figure 12] 10 is a flowchart showing a first reference example of a subroutine of step S8 shown in FIG. 8. [Figure 13] 11 is a diagram showing the positional relationship between a virtual viewpoint and a specific workpiece. FIG. [Figure 14] 11 is a diagram showing an example of a model image corresponding to the posture of a specific workpiece when viewed from a virtual viewpoint. FIG. [Figure 15]13 is a diagram showing an example in which the posture of a specific workpiece is erroneously recognized; FIG. [Figure 16] FIG. 16 is a diagram showing an example of a first model image obtained from the erroneous recognition result shown in FIG. 15. [Figure 17] FIG. 13 is a diagram showing an example of a recognition result recorded in a second reference example of a subroutine in step S8. [Figure 18] 10 is a flowchart showing a second reference example of a subroutine of step S8 shown in FIG. 8. [Figure 19] 10 is a flowchart showing a second embodiment of the subroutine of step S8 shown in FIG. 8. [Figure 20] 10 is a flowchart showing a third embodiment of the subroutine of step S8 shown in FIG. 8. [Figure 21] 9 is a flowchart showing a first embodiment of a subroutine of step S9 shown in FIG. 8. [Figure 22] 22 is a flowchart showing a first embodiment of a subroutine of step S909 shown in FIG. 21. [Diagram 23] 22 is a flowchart showing a second embodiment of the subroutine of step S909 shown in FIG. 21. [Figure 24] 10 is a flowchart showing a second embodiment of the subroutine of step S9 shown in FIG. 8. [Diagram 25] 25 is a flowchart showing a second embodiment of the subroutine of step S936 shown in FIG. 24. [Figure 26] 13 is a flowchart showing an example of a processing flow of a system according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and the description thereof will not be repeated.

[0025] §1 Examples of application An example of a situation in which the present invention is applied will be described with reference to FIG. 1. FIG. 1 is a schematic diagram showing the overall configuration of a system according to an embodiment. The system 1 measures the three-dimensional shapes of a plurality of workpieces 2 randomly piled on a tray 3. The measurement results can be used for various purposes. In the example shown in FIG. 1, the system 1 performs pick-and-place for one target workpiece among the plurality of workpieces 2 based on the measurement results. However, examples of the use of the measurement results of the three-dimensional shape are not limited to pick-and-place.

[0026] As shown in FIG. 1, the system 1 includes an image processing device 100, a three-dimensional camera 200, a robot 300, and a robot controller 400.

[0027] The three-dimensional camera 200 captures an image of a subject including a plurality of workpieces 2 randomly stacked on a tray 3. The three-dimensional camera 200 includes a projection unit 201 and an image capture unit 202. The projection unit 201 projects an arbitrary projection pattern light onto the subject in accordance with a projection instruction from the image processing device 100. The projection pattern is, for example, a pattern in which brightness changes periodically along a predetermined direction within an irradiation surface. The image capture unit 202 captures an image of the subject with the projection pattern light projected thereon in accordance with an image capture instruction from the image processing device 100.

[0028] The projection unit 201 includes, as main components, a light source such as an LED (Light Emitting Diode) or a halogen lamp, and a filter arranged on the side of the irradiation surface of the projection unit 201. The filter generates a projection pattern light required for measuring a three-dimensional shape as described later, and can arbitrarily change the light transmittance in the surface according to an instruction from the image processing device 100. The projection unit 201 may project a fixed projection pattern light that does not change over time, or may project a projection pattern light that changes over time. The projection unit 201 may be configured using a liquid crystal or a DMD (Digital Mirror Device) and a light source (such as an LED or laser light source).

[0029] The photographing unit 202 includes, as main components, an optical system such as a lens, and an imaging element such as a CCD (Coupled Charged Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0030] The robot 300 performs pick-and-place for the workpiece 2. The robot 300 includes a robot arm 301 and a base 302 that supports the robot arm 301. A three-dimensional camera 200 and an end effector 304 are attached to the tip of the robot arm 301. The end effector 304 is configured to hold the workpiece 2, and includes, for example, a suction pad or a two-fingered hand. The operation of the robot 300 is controlled by a robot controller 400.

[0031] The robot controller 400 controls the robot arm 301 according to a command from the image processing device 100. The command indicates, for example, the position and orientation of the tip of the robot arm 301 (hereinafter referred to as the "tool center point"). The robot controller 400 controls the robot arm 301 so that the position and orientation of the tool sensor point match the command.

[0032] The image processing device 100 performs image processing on the captured image received from the three-dimensional camera 200. For example, the image processing device 100 may determine the position and orientation to be taken by the tool center point of the robot arm 301 based on the image processing result. The image processing device 100 may generate a command to move the tool center point to the determined position and orientation, and output the command to the robot controller 400.

[0033] The image processing device 100 includes a three-dimensional measurement unit 10, a viewpoint conversion unit 11, a missing area calculation unit 12, and a determination unit 13.

[0034] The three-dimensional measuring unit 10 generates a real depth map based on the captured images received from the three-dimensional camera 200. The real depth map represents the three-dimensional shape of the subject of the three-dimensional camera 200. The depth map is also called a distance image, and represents the distance from the three-dimensional camera 200 for each pixel.

[0035] A viewpoint conversion unit 11 converts a real depth map into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints. The virtual depth map indicates, for each pixel, the distance from the corresponding virtual viewpoint.

[0036] The missing area calculation unit 12 is an example of a "calculation unit" in the present disclosure, and calculates the area of ​​a missing region where distance information is missing for each of a plurality of virtual depth maps.

[0037] The determination unit 13 determines one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area of ​​a missing region among the multiple virtual depth maps as one or more next imaging position candidates for the three-dimensional camera 200. For example, the determination unit 13 extracts a predetermined number of virtual depth maps having the top areas of missing regions from among the multiple virtual depth maps, and determines the virtual viewpoints corresponding to each of the extracted virtual depth maps as imaging position candidates.

[0038] Thus, according to this embodiment, the next one or more imaging position candidates of the 3D camera 200 are determined from among the multiple virtual viewpoints, taking into account the area of ​​the missing area. The missing area of ​​each virtual depth map corresponds to an area where occlusion occurs when viewed from the latest imaging position corresponding to the captured image used to generate the real depth map. This allows the image processing device 100 to determine imaging position candidates suitable for measuring the three-dimensional shape of the area where occlusion occurs.

[0039] §2 Specific examples <Example of hardware configuration of image processing device> The image processing device 100 is typically a computer having a general-purpose architecture, and executes a pre-installed program (instruction code) to perform the image capture position candidate determination process according to this embodiment. Such a program is typically distributed in a state stored in various recording media, or is installed in the image processing device 100 via a network, etc.

[0040] When such a general-purpose computer is used, an OS (Operating System) for executing basic processing of the computer may be installed in addition to an application for executing the process of determining the imaging position candidate according to the present embodiment. In this case, the program according to the present embodiment may call necessary modules among program modules provided as part of the OS in a predetermined sequence at a predetermined timing to execute the processing. That is, the program according to the present embodiment itself may not include the above-mentioned modules, and may execute the processing in cooperation with the OS. The program according to the present embodiment may be in a form that does not include some of these modules.

[0041] Furthermore, the program according to the present embodiment may be provided by being incorporated into a part of another program. In this case, the program itself does not include the modules included in the other program to be combined as described above, and the processing is executed in cooperation with the other program. That is, the program according to the present embodiment may be in a form incorporated into such other program. Note that some or all of the functions provided by the execution of the program may be implemented as a dedicated hardware circuit.

[0042] Fig. 2 is a schematic diagram showing an example of a hardware configuration of the image processing device shown in Fig. 1. As shown in Fig. 2, the image processing device 100 includes a CPU (Central Processing Unit) 110, which is an example of a processor, a main memory 112, a storage 114, a camera interface 116, an input interface 118, a display controller 120, a communication interface 124, and a data reader / writer 126. These components are connected to each other via a bus 128 so as to be able to communicate data with each other.

[0043] The CPU 110 loads the programs 115 installed in the storage 114 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), and holds images acquired from the three-dimensional camera 200 in addition to the programs 115 read from the storage 114. Furthermore, the storage 114 is a non-volatile storage device such as a solid state drive (SSD) or a hard disk drive (HDD). The storage 114 stores various data, which will be described later. The storage 114 may include a semiconductor storage device such as a flash memory.

[0044] The camera interface 116 mediates data transmission between the CPU 110 and the three-dimensional camera 200. That is, the camera interface 116 is connected to the three-dimensional camera 200. The camera interface 116 provides projection commands and image capture commands to the three-dimensional camera 200 in accordance with internal commands generated by the CPU 110. The camera interface 116 includes an image buffer 116a for temporarily storing images from the three-dimensional camera 200. When a predetermined number of images are stored in the image buffer 116a, the camera interface 116 transfers the stored images to the main memory 112.

[0045] The input interface 118 mediates data transmission between the CPU 110 and the input device 160. That is, the input interface 118 accepts input information input by a user to the input device 160. The input device 160 is an example of a "user interface" in the present disclosure.

[0046] The display controller 120 is connected to a display device 150, and controls the screen of the display device 150 so as to notify the user of the processing results of the CPU 110, etc.

[0047] The communication interface 124 mediates data transmission between the CPU 110 and an external device such as the robot controller 400. The communication interface 124 typically includes Ethernet (registered trademark) or USB (Universal Serial Bus), or the like.

[0048] The data reader / writer 126 mediates data transmission between the CPU 110 and the memory card 106, which is a recording medium. That is, the memory card 106 is distributed in a state in which a program to be executed by the image processing device 100 and the like are stored, and the data reader / writer 126 reads the program from the memory card 106. In addition, in response to an internal command from the CPU 110, the data reader / writer 126 writes images captured by the three-dimensional camera 200 and / or processing results in the image processing device 100 to the memory card 106. The memory card 106 includes a general-purpose semiconductor storage device such as an SD (Secure Digital), a magnetic storage medium such as a flexible disk, or an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory).

[0049] <Example of robot controller hardware configuration> 3 is a schematic diagram showing an example of the hardware configuration of a robot controller according to this embodiment. As shown in FIG. 3, the robot controller 400 includes a network controller 410 and a control processing circuit 420.

[0050] The network controller 410 mainly exchanges data with the image processing device 100 via the network.

[0051] The control processing circuit 420 executes arithmetic processing required to drive the robot 300. As an example, the control processing circuit 420 includes a processor 430, a main memory 440, a storage 450, and an interface circuit 460. The interface circuit 460 exchanges data with the robot 300.

[0052] The processor 430 executes control calculations for driving the robot 300. The main memory 440 is configured with, for example, a volatile storage device such as a DRAM or an SRAM. The storage 450 is configured with, for example, a non-volatile storage device such as an SSD or an HDD.

[0053] The storage 450 stores a system program 452 for implementing control for driving the robot 300. The system program 452 includes instructions for executing control calculations related to the operation of the robot 300, and instructions related to the interface with the robot 300.

[0054] <Functional configuration> Fig. 4 is a diagram showing an example of the functional configuration of the image processing device and the robot controller. As shown in Fig. 4, the image processing device 100 includes a three-dimensional measuring unit 10, a viewpoint conversion unit 11, a missing area calculation unit 12, a determination unit 13, a model image generation unit 14, an assignment unit 15, a three-dimensional recognition unit 17, a recognition result storage unit 18, a picking position calculation unit 19, a range setting unit 20, a priority setting unit 21, a registration unit 22, a candidate list management unit 23, and a communication unit 24.

[0055] The three-dimensional measurement unit 10, viewpoint conversion unit 11, missing area calculation unit 12, determination unit 13, model image generation unit 14, assignment unit 15, three-dimensional recognition unit 17, picking position calculation unit 19, range setting unit 20, priority setting unit 21, registration unit 22, and candidate list management unit 23 are realized by the CPU 110 shown in Fig. 2 executing a program 115. The recognition result storage unit 18 is configured by the main memory 112 or the storage 114. The communication unit 24 is configured by a communication interface 124.

[0056] The three-dimensional measuring unit 10 outputs a projection instruction and an image capturing instruction to the three-dimensional camera 200 in response to the three-dimensional camera 200 reaching the next image capturing position. As a result, the three-dimensional camera 200 captures the multiple workpieces 2 onto which the projection pattern is projected. The three-dimensional measuring unit 10 generates a real depth map based on the captured image received from the three-dimensional camera 200. The three-dimensional measuring unit 10 generates a real depth map using, for example, a triangulation method. As the triangulation method, known techniques such as a fixed pattern projection method, a phase shift method, a stereo matching method, and a light section method can be adopted. For example, the phase shift method is a method that uses an image captured in a state where a projection pattern light in which the shading is changed sinusoidally is projected onto a subject. A plurality of projection patterns with different shading change periods and phases are prepared, and the projection unit 201 of the three-dimensional camera 200 sequentially irradiates the multiple projection pattern lights onto the field of view of the image capturing unit 202. The three-dimensional measurement unit 10 measures the three-dimensional shape of the visual field area based on the change in brightness (luminance or brightness) in the captured image when multiple projection patterns are projected, and generates a real depth map indicating the measurement results. Note that the three-dimensional measurement process is not limited to the triangulation method, and may be performed using other methods (such as a coaxial method).

[0057] The model image generation unit 14 generates a model image set 16 including a plurality of model images 60 (see FIG. 7) showing a work model corresponding to the work 2, based on CAD (Computer Aided Design) data showing the three-dimensional shape of the work 2.

[0058] 5 is a diagram for explaining a method for generating a model image. The model image generating unit 14 places a work model 2M having a three-dimensional shape represented by CAD data in a virtual space. Furthermore, the model image generating unit 14 sets a plurality of points P on a spherical surface centered on the work model 2M. The distance between the plurality of points P and the work model 2M is constant. The model image generating unit 14 generates a depth map of the work model 2M when viewed from each of the plurality of points P as a model image 60.

[0059] 4, every time a real depth map is generated, the three-dimensional recognition unit 17 executes an object recognition process to recognize the position and orientation of the workpiece 2 from the real depth map by template matching using a plurality of model images 60. The three-dimensional recognition unit 17 stores the recognition result of the position and orientation of the workpiece 2 in the recognition result storage unit 18.

[0060] Template matching is a technique for searching for an area similar to a model image in a target image (a real depth map in this embodiment). The three-dimensional recognition unit 17 calculates feature amounts from each of the real depth map and the model image 60, and calculates a score (hereinafter referred to as a "matching score") that indicates the degree of match between the feature amounts. The three-dimensional recognition unit 17 recognizes the position and orientation of the workpiece 2 in the real depth map by searching for an area similar to the model image 60 in the real depth map based on the matching score.

[0061] Fig. 6 is a diagram showing an example of the recognition result stored in the recognition result storage unit. As shown in Fig. 6, the recognition result indicates, for example, for each workpiece 2, the recognized position and orientation, and a model image ID that identifies the model image having the highest matching score with the area including the workpiece 2 in the actual depth map.

[0062] 4, the assigning unit 15 assigns a score (hereinafter referred to as a "recognition ease score") representing the recognizability of the posture of the workpiece 2 in template matching to each of the multiple model images 60. The model image set 16 includes the recognizability score assigned to each model image 60.

[0063] Fig. 7 is a diagram showing an example of a model image set 16. As shown in Fig. 7, the model image set 16 includes a plurality of model images 60 to which recognizability scores have been assigned.

[0064] The recognizability is determined based on the uniqueness of the model image 60 and the area (size) of the work model 2M in the model image 60.

[0065] For example, when the work model 2M is spherical, the shape of the work model 2M viewed from multiple points P shown in FIG. 5 is substantially the same. In this case, the uniqueness of each model image 60 is small. When a model image 60 with low uniqueness is used, the three-dimensional recognition unit 17 may not be able to correctly recognize the posture of the work 2. Therefore, the assignment unit 15 evaluates the ease of recognition based on the uniqueness of the model image 60. The uniqueness of a certain model image 60 is greater the lower the similarity with other model images 60. Therefore, the assignment unit 15 calculates a score (hereinafter referred to as a "uniqueness score") that represents the degree of deviation between the feature amount calculated from the target model image 60 and the feature amount calculated from the other model images 60 as an index representing the uniqueness of the target model image 60.

[0066] When the area of ​​the work model 2M in the model image 60 is large, the feature amount calculated from the model image 60 better represents the features of the work model 2M. Therefore, when a model image 60 in which the area of ​​the work model 2M is large is used, the three-dimensional recognition unit 17 can more easily correctly recognize the posture of the work 2. Therefore, the assignment unit 15 evaluates the ease of recognition based on the area of ​​the work model 2M in the model image 60. The area of ​​the work model 2M is represented, for example, by the number of pixels in which the work model 2M exists in the model image 60.

[0067] The assigning unit 15 calculates the recognizability score by inputting the uniqueness score of each model image 60 and the area of ​​the work model 2M into a predetermined relational expression. The relational expression is determined so that the larger the uniqueness score, the larger the recognizability score, and the larger the area of ​​the work model 2M, the larger the recognizability score.

[0068] The assigning unit 15 may assign a recognizability score to each model image 60 according to a user input. In this case, the user may check a plurality of model images 60 and input a recognizability score for each model image 60 in consideration of the recognizability of the work 2 in template matching.

[0069] 4, the picking position calculation unit 19 selects a workpiece 2 to be picked by the robot arm 301 (hereinafter referred to as a "workpiece to be picked") based on the latest recognition result stored in the recognition result storage unit 18. The picking position calculation unit 19 calculates the position and orientation of the tool center point when picking up the workpiece to be picked (hereinafter referred to as a "picking position and orientation").

[0070] The picking position calculation unit 19 specifies the position and orientation of the workpiece to be picked from the recognition result. For example, when the recognition result indicates the positions and orientations of multiple works 2, the picking position calculation unit 19 specifies the workpiece 2 having the position and orientation with the highest matching score as the workpiece to be picked.

[0071] The picking position calculation unit 19 calculates the picking position and posture so that the relative position and posture of the end effector 304 with respect to the workpiece to be picked becomes a pre-registered position and posture (hereinafter referred to as the "registered position and posture"). The registered position and posture are registered in advance based on, for example, CAD data indicating the shape of the workpiece 2 and CAD data indicating the shape of the end effector 304.

[0072] The picking position calculation unit 19 generates a pick-and-place command to which the picking position and orientation are added, and outputs the pick-and-place command to the robot controller 400 via the communication unit 24.

[0073] The viewpoint conversion unit 11 converts the real depth map into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints. The plurality of virtual viewpoints are arranged on a spherical surface so that the distance from a reference point (e.g., the center of the tray 3) is constant. The plurality of virtual viewpoints are arranged, for example, at equal intervals on the spherical surface.

[0074] The range setting unit 20 sets the movable range of the three-dimensional camera in response to an input to the input device 160. When the movable range is set by the range setting unit 20, the viewpoint conversion unit 11 places a plurality of virtual viewpoints within the movable range.

[0075] For example, the user can set the movable range so as to exclude positions that are obviously inappropriate for photographing the workpieces 2 randomly piled on the tray 3, depending on the position of the tray 3. This prevents a position that is obviously inappropriate for photographing the workpieces 2 from being selected as the virtual viewpoint. If a position that is obviously inappropriate for photographing the workpieces 2 is selected as the virtual viewpoint, the generation of a virtual depth map corresponding to the virtual viewpoint and the processing of the virtual depth map corresponding to the virtual viewpoint are unnecessary. Therefore, by setting the movable range, the time required for such unnecessary processing can be reduced.

[0076] The missing area calculation unit 12 calculates the area of ​​a missing region where distance information is missing for each of the multiple virtual depth maps.

[0077] The determination unit 13 determines one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area of ​​the missing region among the multiple virtual depth maps as one or more next imaging position candidates for the three-dimensional camera 200. Furthermore, the determination unit 13 may narrow down the one or more imaging position candidates based on the recognizability of the posture of the workpiece 2 when viewed from the virtual viewpoint. The recognizability is evaluated based on the recognizability score assigned to the model image 60 corresponding to the estimated posture of the workpiece 2 when viewed from the virtual viewpoint.

[0078] When an inappropriate position is registered by the registration unit 22, the determination unit 13 determines a position other than the inappropriate position as the imaging position candidate.

[0079] The registration unit 22 registers the inappropriate position. For example, the registration unit 22 registers the latest image capture position as an inappropriate position in response to receiving a notification from the robot controller 400 indicating that picking has failed. Furthermore, the registration unit 22 may register the inappropriate position based on the recognition result stored in the recognition result storage unit 18. A method of registering an inappropriate position based on the recognition result will be described later in detail.

[0080] The priority setting unit 21 sets a priority for each of the one or more imaging position candidates determined by the determination unit 13 based on one or more indices. The one or more indices include, for example, the area of ​​the missing area. The one or more indices may include a movement amount of the robot arm 301 for moving the three-dimensional camera to the imaging position candidate. The movement amount is calculated according to the difference between the position and orientation of the robot arm 301 after pick-and-place and the position and orientation of the robot arm 301 when the three-dimensional camera 200 is located at the imaging position candidate. The one or more indices may include the recognizability of the workpiece 2 when viewed from the imaging position candidate. The recognizability is evaluated based on a recognizability score assigned to the model image 60 corresponding to the estimated orientation of the workpiece 2 when viewed from the imaging position candidate.

[0081] The candidate list management unit 23 is an example of an "output unit" in the present disclosure. The candidate list management unit 23 manages a list of one or more imaging position candidates. Furthermore, the candidate list management unit 23 manages a priority set for each imaging position candidate. The candidate list management unit 23 generates an inquiry as to whether the three-dimensional camera 200 can move to the imaging position candidate, and outputs the inquiry to the robot controller 400 via the communication unit 24.

[0082] The candidate list management unit 23 may select one imaging position candidate from the list in descending order of priority and generate an inquiry regarding the selected imaging position candidate. In this case, the candidate list management unit 23 selects the imaging position candidate with the highest priority from the list and generates an inquiry regarding the selected imaging position candidate. Thereafter, the candidate list management unit 23 receives a notification indicating the determination result of whether or not movement is possible from the robot controller 400 via the communication unit 24. If the determination result indicates that movement is not possible, the candidate list management unit 23 may delete from the list the imaging position candidate to which the three-dimensional camera 200 cannot move, and generate a new inquiry regarding the imaging position candidate with the highest priority among the remaining imaging position candidates. If the determination result indicates that movement is possible, the candidate list management unit 23 stops generating a new inquiry.

[0083] Alternatively, the candidate list management unit 23 may generate an inquiry regarding all of one or more imaging position candidates included in the list. In this case, the candidate list management unit 23 receives a notification indicating the result of the determination as to whether movement is possible from the robot controller 400 via the communication unit 24. The notification indicates that movement is possible and indicates the imaging position candidate determined as the next imaging position.

[0084] If the three-dimensional camera 200 cannot be moved to all of the one or more imaging position candidates, the candidate list management unit 23 may determine a position other than the one or more imaging position candidates as the next imaging position. In this case, the candidate list management unit 23 generates a movement command for moving the three-dimensional camera 200 to the next imaging position, and outputs the movement command to the robot controller 400 via the communication unit 24.

[0085] The communication unit 24 transmits and receives data to and from an external device. Specifically, the communication unit 24 transmits a pick-and-place command generated by the picking position calculation unit 19 to the robot controller 400. The communication unit 24 transmits an inquiry or a movement command generated by the candidate list management unit 23 to the robot controller 400. Furthermore, the communication unit 24 receives from the robot controller 400 a notification indicating that the three-dimensional camera 200 has reached the next imaging position, a notification indicating that picking has failed, or a notification indicating the result of a determination as to whether movement is possible or not.

[0086] As shown in Fig. 4, the robot controller 400 includes a communication unit 41, a movement permission determination unit 42, and a control unit 43. The communication unit 41 is configured by the network controller 410 shown in Fig. 3. The movement permission determination unit 42 and the control unit 43 are configured by a control processing circuit 420.

[0087] The communication unit 41 transmits and receives data to and from an external device. Specifically, the communication unit 41 receives a pick-and-place command, an inquiry, or a movement command from the image processing device 100. When the three-dimensional camera 200 reaches the next imaging position, the communication unit 41 transmits a notification indicating that the three-dimensional camera 200 has reached the next imaging position to the image processing device 100. When the robot 300 fails to pick up, the communication unit 41 transmits a notification indicating that picking has failed to the image processing device 100. Furthermore, the communication unit 41 transmits a notification indicating the result of the determination made by the movement feasibility determination unit 42 to the image processing device 100.

[0088] The movement feasibility determination unit 42 determines whether or not the three-dimensional camera 200 can be moved to the imaging position candidate in accordance with the inquiry received from the image processing device 100. The movement feasibility determination unit 42 determines whether or not the three-dimensional camera 200 can be moved to the imaging position candidate based on the kinematics of the robot 300. The movement feasibility determination unit 42 outputs the determination result to the image processing device 100 via the communication unit 41.

[0089] The movement possibility determination unit 42 determines a candidate imaging position to which the three-dimensional camera 200 can move as the next imaging position, and outputs a movement command to the control unit 43 to move the three-dimensional camera 200 to the determined imaging position.

[0090] The control unit 43 controls the operation of the robot 300. Specifically, the control unit 43 controls the robot 300 to perform a series of operations in response to a pick-and-place command, such as picking up the workpiece 2 to be picked and placing the workpiece in a predetermined location.

[0091] The control unit 43 monitors the state of the end effector 304 and determines whether the picking operation has been successful or not according to the state of the end effector 304. In response to determining that picking has failed, the control unit 43 generates a notification indicating that picking has failed and outputs the notification to the image processing device 100 via the communication unit 41.

[0092] Furthermore, the control unit 43 controls the robot 300 so that the three-dimensional camera 200 moves to the next imaging position in response to a movement command received from the image processing device 100 or the movement feasibility determination unit 42. In response to the three-dimensional camera 200 reaching the next imaging position, the control unit 43 generates a notification indicating that the three-dimensional camera 200 has reached the next imaging position. The control unit 43 outputs the generated notification to the image processing device 100 via the communication unit 41.

[0093] <System processing flow> Fig. 8 is a flowchart showing an example of the flow of processing of the system shown in Fig. 1. As shown in Fig. 8, first, in step S1, the CPU 110 operating as the range setting unit 20 of the image processing device 100 sets the movable range of the three-dimensional camera 200 in response to a user input to the input device 160.

[0094] In the next step S2, the CPU 110 outputs a command to the robot controller 400 to move the three-dimensional camera 200 to a predetermined initial position. The initial position is, for example, vertically above the center of the tray 3. The control processing circuit 420 operating as the control unit 43 of the robot controller 400 controls the robot 300 so that the three-dimensional camera 200 moves to the initial position. As a result, the three-dimensional camera 200 moves to the initial position.

[0095] In the next step S3, the CPU 110 operating as the three-dimensional measuring unit 10 of the image processing device 100 outputs a projection instruction and an image capturing instruction to the three-dimensional camera 200. As a result, the three-dimensional camera 200 captures an image of the subject (which may include the workpieces 2 randomly piled on the tray 3) onto which the projection pattern is projected.

[0096] In the next step S4, CPU 110 measures the three-dimensional shape of the subject based on the captured image obtained by three-dimensional camera 200. CPU 110 generates a real depth map indicating the three-dimensional shape of the subject.

[0097] In the next step S5, the CPU 110 operating as the three-dimensional recognition unit 17 of the image processing device 100 performs object recognition processing to recognize the position and orientation of the workpiece 2 from the real depth map by template matching using a plurality of model images 60. The CPU 110 records the recognition result as shown in FIG.

[0098] In the next step S6, the CPU 110 operating as the picking position calculation unit 19 of the image processing device 100 judges whether or not the positions and orientations of one or more workpieces 2 have been recognized. Specifically, the CPU 110 judges whether or not the latest recognition result indicates the positions and orientations of one or more workpieces 2.

[0099] When the position and orientation of one or more workpieces 2 are recognized (YES in step S6), the system 1 executes step S7. In step S7, the CPU 110 selects a workpiece to be picked from the one or more workpieces 2 whose positions and orientations have been recognized. The CPU 110 calculates the position and orientation (picking position and orientation) of the tool center point when picking the workpiece to be picked, and outputs a pick-and-place command to which the picking position and orientation has been added, to the robot controller 400. The control processing circuit 420 operating as the control unit 43 of the robot controller 400 controls the robot 300 to pick and place the workpiece to be picked in response to the pick-and-place command.

[0100] If the position and orientation of one or more workpieces 2 cannot be recognized (NO in step S6), the system 1 ends the process.

[0101] In step S8 after step S7, the CPU 110 of the image processing device 100 determines one or more next imaging position candidates for the three-dimensional camera 200.

[0102] In the next step S9, the system 1 determines the next imaging position of the 3D camera 200.

[0103] In the next step S10, the control processing circuit 420 operating as the control unit 43 of the robot controller 400 controls the robot 300 to move the three-dimensional camera 200 to the next imaging position. After step S10, the system 1 returns the process to step S3.

[0104] <First Example of Subroutine of Step S8> A first embodiment of the subroutine of step S8 will be described with reference to Figures 9 to 11. Figure 9 is a flowchart showing the process flow of a first embodiment of the subroutine of step S8 shown in Figure 8. As shown in Figure 9, the first embodiment of the subroutine of step S8 includes steps S801 to S804.

[0105] First, in step S801, the CPU 110 operating as the viewpoint conversion unit 11 of the image processing device 100 obtains a real depth map corresponding to the latest imaging position.

[0106] In the next step S802, the CPU 110 converts the real depth map into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints.

[0107] In the next step S803, the CPU 110 operating as the missing area calculation unit 12 of the image processing device 100 calculates the area of ​​a missing region where distance information is missing for each of the multiple virtual depth maps.

[0108] In the next step S804, the CPU 110 operating as the determination unit 13 of the image processing device 100 determines one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area of ​​the missing region as one or more next imaging position candidates for the three-dimensional camera 200. For example, the CPU 110 extracts a predetermined number of virtual depth maps having the top areas of the missing regions from among the multiple virtual depth maps, and determines the virtual viewpoints corresponding to each of the extracted virtual depth maps as the imaging position candidates.

[0109] Fig. 10 is a diagram showing an example of a plurality of virtual viewpoints, and Fig. 11 is a diagram showing an example of a virtual depth map viewed from the virtual viewpoints shown in Fig. 10.

[0110] As shown in FIG. 10, the CPU 110 arranges a plurality of virtual viewpoints VP so that the distance from a reference point OP (e.g., the center of the tray 3) coincides with the distance L. In the example shown in FIG. 10, the plurality of virtual viewpoints VP include virtual viewpoints VP_1 and VP_2. The distance L is the distance between the latest imaging position RP of the three-dimensional camera 200 and the reference point OP. As shown in FIG. 11, the CPU 110 converts the real depth map 50 into a virtual depth map 52 seen from each of the plurality of virtual viewpoints VP. In FIG. 11, the virtual depth map 52_1 is a virtual depth map seen from the virtual viewpoint VP_1. The virtual depth map 52_2 is a virtual depth map seen from the virtual viewpoint VP_2.

[0111] In the example shown in FIG. 10, workpieces 2_1 and 2_2 having an L-shaped cross section are present on the tray 3. The three-dimensional camera 200 located at the latest imaging position RP receives reflected light from the surfaces 2a and 2b of the workpiece 2_1 and reflected light from the surfaces 2c and 2d of the workpiece 2_2. Furthermore, the three-dimensional camera 200 receives reflected light from areas 3a, 3b, and 3c of the upper surface of the tray 3. Therefore, the CPU 110 can accurately measure the three-dimensional shapes of the surfaces 2a and 2b of the workpiece 2_1, the surfaces 2c and 2d of the workpiece 2_2, and the areas 3a, 3b, and 3c of the upper surface of the tray 3 based on the captured image received from the three-dimensional camera 200.

[0112] The three-dimensional camera 200 can receive reflected light from a surface 2z of the workpiece 2_1. However, the surface 2z is approximately parallel to an optical axis 250 of the three-dimensional camera 200. Therefore, the CPU 110 cannot measure the three-dimensional shape of the surface 2z.

[0113] When viewed from the three-dimensional camera 200 located at the latest imaging position RP, the area 3z on the upper surface of the tray 3 is hidden by the workpiece 2_1. Therefore, the CPU 110 cannot measure the three-dimensional shape of the area 3z.

[0114] As shown in Fig. 10, the region 3z is visible from the virtual viewpoint VP_1. However, the real depth map does not have distance information for the region 3z. Therefore, as shown in Fig. 11, the virtual depth map 52_1 corresponding to the virtual viewpoint VP_1 has a missing region 56_1 in which distance information is missing. The missing region 56_1 corresponds to the region 3z.

[0115] As shown in Fig. 10, the surface 2z of the workpiece 2_1 is visible from the virtual viewpoint VP_2. Also, when viewed from the virtual viewpoint VP_2, the region 3z is located on the back side of the surface 2z. However, the real depth map does not have distance information for the surface 2z and the region 3z. Therefore, as shown in Fig. 11, the virtual depth map 52_2 corresponding to the virtual viewpoint VP_2 has a missing region 56_2 where distance information is missing. The missing region 56_2 corresponds to the surface 2z (and the region 3z on the back side of the surface 2z).

[0116] The CPU 110 may calculate the areas of the defective regions 56_1 and 56_2, and determine the virtual viewpoint VP corresponding to the virtual depth map 52 having a relatively large area as the imaging position candidate.

[0117] 11, the virtual depth maps 52_1 and 52_2 may include an outside-frame region 54 outside a virtual frame line 53 that corresponds to the image frame of the real depth map 50. The outside-frame region 54 corresponds to the outside of the field of view of the three-dimensional camera 200 located at the imaging position RP that corresponds to the real depth map 50. Therefore, the CPU 110 does not include the outside-frame region 54 in the missing region.

[0118] The virtual depth maps 52_1 and 52_2 may include minute regions in which distance information is missing due to the influence of noise, etc. In order not to count such minute regions as missing regions, it is preferable that the CPU 110 determines, as missing regions, regions in which a predetermined number or more of pixels in which distance information is missing are connected.

[0119] <First Reference Example of Subroutine of Step S8> Fig. 12 is a flow chart showing a first reference example of a subroutine of step S8 shown in Fig. 8. As shown in Fig. 12, the first reference example of the subroutine of step S8 includes steps S811 and S812.

[0120] First, in step S811, the CPU 110 operating as the determination unit 13 of the image processing device 100 acquires a recognition result corresponding to the latest real depth map (hereinafter, referred to as the "latest recognition result").

[0121] In the next step S812, the CPU 110 determines, based on the latest recognition result, one or more virtual viewpoints that increase the ease of recognition of one or more specific works as one or more imaging position candidates.

[0122] Details of step S812 will be described with reference to Fig. 13 and Fig. 14. Fig. 13 is a diagram showing the positional relationship between a virtual viewpoint and a specific workpiece. Fig. 14 is a diagram showing an example of a model image corresponding to the posture of the specific workpiece when viewed from the virtual viewpoint.

[0123] The CPU 110 identifies one or more workpieces 2 other than the workpiece to be picked as one or more specific workpieces 2S. For example, the CPU 110 identifies a workpiece 2 having a position and orientation corresponding to the highest matching score among one or more workpieces 2 other than the workpiece to be picked as the specific workpiece 2S. Alternatively, the CPU 110 identifies a workpiece 2 having a position and orientation that is in the top predetermined number of matching scores among one or more workpieces 2 other than the workpiece to be picked as the specific workpiece 2S.

[0124] The CPU 110 determines one or more virtual viewpoints VP that satisfy a specific condition as one or more candidate imaging positions for the three-dimensional camera 200. The specific condition is a condition that a first recognizability of the posture of one or more specific workpieces 2S when viewed from the virtual viewpoint VP is higher than a second recognizability of the posture of one or more specific workpieces 2S in the real depth map.

[0125] The first recognizability is determined based on the recognizability score of a model image (hereinafter referred to as the “first model image 61”) among the multiple model images 60 that corresponds to the estimated posture of each of one or more specific works 2S when viewed from the virtual viewpoint VP.

[0126] The posture of each specific work 2S when viewed from the virtual viewpoint VP is estimated based on the posture of the specific work 2S recognized from the real depth map, the virtual viewpoint VP, and the latest imaging position RP (imaging position corresponding to the latest real depth map) of the three-dimensional camera 200. Specifically, the posture of each specific work 2S when viewed from the virtual viewpoint VP is estimated by applying a rotation transformation about a reference point OP to the recognized posture of the specific work 2S in order to match the virtual viewpoint VP with the latest imaging position RP.

[0127] The second recognizability is determined based on the recognizability score of a model image (hereinafter referred to as the “second model image 62”) among the multiple model images 60 that corresponds to the recognized posture of each of the one or more specific works 2S.

[0128] 13, the CPU 110 identifies one specific work 2S. The CPU 110 identifies, as the second model image 62, the model image having the highest matching score with the area including the specific work 2S in the actual depth map based on the recognition result shown in FIG.

[0129] The CPU 110 reads out the recognizability score corresponding to the second model image 62 from the model image set 16 shown in Fig. 7, and evaluates the second recognizability based on the read recognizability score. When one specific work 2S is identified, the CPU 110 may determine the recognizability score corresponding to the second model image 62 as the evaluation value of the second recognizability. When multiple specific works 2S are identified, the CPU 110 may identify multiple second model images 62 corresponding to the multiple specific works 2S, and determine a representative value (e.g., average value, median value, etc.) of the multiple recognizability scores corresponding to the multiple second model images 62 as the evaluation value of the second recognizability.

[0130] The CPU 110 estimates the posture of the specific work 2S as viewed from the virtual viewpoint VP_1 based on the recognized posture of the specific work 2S, the virtual viewpoint VP_1, and the latest imaging position RP, and identifies the first model image 61_1 corresponding to the estimated posture. The CPU 110 reads out the recognizability score corresponding to the first model image 61_1 from the model image set 16, and evaluates the first recognizability of the specific work 2S as viewed from the virtual viewpoint VP_1 based on the read recognizability score. The method of evaluating the first recognizability is the same as the method of evaluating the second recognizability.

[0131] Similarly, the CPU 110 estimates the posture of the specific work 2S when viewed from the virtual viewpoint VP_2 based on the recognized posture of the specific work 2S, the virtual viewpoint VP_2, and the latest imaging position RP, and identifies the first model image 61_2 corresponding to the estimated posture. The CPU 110 reads out the recognizability score corresponding to the first model image 61_2 from the model image set 16, and evaluates the first recognizability of the specific work 2S when viewed from the virtual viewpoint VP_2 based on the read out recognizability score.

[0132] 13 and 14, the uniqueness of the first model image 61_1 is greater than the uniqueness of the second model image 62. Therefore, the recognizability score corresponding to the first model image 61_1 is greater than the recognizability score corresponding to the second model image 62. Therefore, the CPU 110 determines that the virtual viewpoint VP_1 satisfies the above-mentioned specific condition, and determines the virtual viewpoint VP_1 as the next imaging position candidate.

[0133] The area of ​​the work model 2M in the first model image 61_2 is larger than the area of ​​the work model 2M in the second model image 62. Therefore, the recognizability score corresponding to the first model image 61_2 is larger than the recognizability score corresponding to the second model image 62. Therefore, the CPU 110 determines that the virtual viewpoint VP_2 satisfies the above specific condition, and determines the virtual viewpoint VP_2 as the next imaging position candidate.

[0134] <Second Reference Example of Subroutine of Step S8> The second reference example of the subroutine of step S8 is a modification of the above-mentioned first reference example. In the object recognition process, the posture of the specific workpiece 2S may be erroneously recognized.

[0135] 15 is a diagram showing an example in which the posture of a specific workpiece 2S is erroneously recognized. In the example shown in FIG.

[0136] FIG. 16 is a diagram showing an example of a first model image obtained from the erroneous recognition result shown in FIG. 15. In FIG. 16, the first model images 61_3, 61_4 and the second model image 62_1 are identified by applying the erroneous recognition result shown in FIG. 15 to the first reference example of the subroutine of step S8 described above. That is, the second model image 62_1 corresponds to the erroneously recognized posture. The first model image 61_3 corresponds to the estimated posture of the specific workpiece 2S when viewed from the virtual viewpoint VP_1. The first model image 61_4 corresponds to the estimated posture of the specific workpiece 2S when viewed from the virtual viewpoint VP_2.

[0137] The recognizability score corresponding to the first model image 61_3 is greater than the recognizability score corresponding to the second model image 62_1. On the other hand, the recognizability score corresponding to the first model image 61_4 is less than the recognizability score corresponding to the second model image 62_1. Therefore, the virtual viewpoint VP_1 is determined to satisfy the above specific condition, and is determined as the next imaging position candidate.

[0138] When the virtual viewpoint VP_1 is determined as the next imaging position and a new real depth map is acquired, the posture of the specific workpiece 2S is recognized from the new real depth map. At this time, if the posture of the specific workpiece 2S is correctly recognized, the model image 60 corresponding to the recognized posture is different from the first model image 61_3. In this way, after the posture of the specific workpiece 2S is erroneously recognized in the k-th object recognition process, the posture of the specific workpiece 2S may be correctly recognized in the k+1-th object recognition process. In this case, the first model image 61 corresponding to the posture of the specific workpiece 2S estimated from the recognition result by the k-th object recognition process does not match the second model image 62 corresponding to the posture of the specific workpiece 2S recognized in the k+1-th object recognition process.

[0139] Conversely, there may be cases where the posture of the specific work 2S is correctly recognized in the k-th object recognition process, and then the posture of the specific work 2S is erroneously recognized in the k+1-th object recognition process. In this case as well, the first model image 61 corresponding to the posture of the specific work 2S estimated from the recognition result of the k-th object recognition process does not match the second model image 62 corresponding to the posture of the specific work 2S recognized in the k+1-th object recognition process.

[0140] In this way, if the first model image 61 corresponding to the posture of the specific work 2S estimated from the recognition result by the kth object recognition process does not match the second model image 62 corresponding to the posture of the specific work 2S recognized in the k+1th object recognition process, the posture of the specific work 2S has been erroneously recognized in one of the object recognition processes. A virtual viewpoint that may erroneously recognize the posture of the specific work 2S is inappropriate as a candidate for the next shooting position. Therefore, the second reference example of the subroutine of step S8 includes a process of registering a virtual viewpoint that may erroneously recognize the posture of the specific work 2S as an inappropriate position.

[0141] In a second reference example of the subroutine of step S8, for each specific workpiece 2S, a first model image 61 corresponding to a posture estimated from the previous recognition result is compared with a second model image 62 corresponding to a posture estimated from the latest recognition result. Therefore, the CPU 110 records a model image ID that identifies the first model image 61 corresponding to the next imaging position.

[0142] Fig. 17 is a diagram showing an example of the recognition result recorded in the second reference example of the subroutine of step S8. As shown in Fig. 17, the CPU 110 records, in the recognition result, a model image ID that identifies the first model image 61 corresponding to the estimated orientation when viewed from the next imaging position for each specific workpiece 2S. In the example shown in Fig. 17, the workpiece 2 with the position and orientation "X2, Y2, Z2, Xr2, Yr2, Zr2" is identified as the specific workpiece 2S, and a model image ID that identifies the first model image 61 for the specific workpiece 2S is recorded.

[0143] When the CPU 110 determines a plurality of imaging position candidates, it records the model image ID that identifies the first model image 61 corresponding to each specific workpiece 2S for each imaging position candidate. After that, when the next imaging position is determined from among the plurality of imaging position candidates, it is sufficient to delete the model image ID that identifies the first model image 61 corresponding to the imaging position candidate other than the next imaging position from the recognition result.

[0144] Fig. 18 is a flow chart showing a second reference example of the subroutine of step S8 shown in Fig. 8. As shown in Fig. 18, the second reference example of the subroutine of step S8 includes steps S812, and S821 to S824.

[0145] First, in step S821, the CPU 110 of the image processing device 100 acquires a recognition result corresponding to the previous real depth map (hereinafter referred to as the "previous recognition result") and a recognition result corresponding to the latest real depth map (hereinafter referred to as the "latest recognition result").

[0146] In the next step S822, the CPU 110 operating as the registration unit 22 of the image processing device 100 determines whether or not the first model image 61 recorded in the previous recognition result matches the second model image 62 recorded in the latest recognition result for each of one or more specific works 2S identified from the previous recognition result.

[0147] If the first model image 61 and the second model image 62 do not match for any of the one or more specific workpieces 2S (NO in step S822), the system 1 advances the process to step S823. If the first model image 61 and the second model image 62 match for all of the one or more specific workpieces 2S (YES in step S822), the system 1 advances the process to step S812.

[0148] In step S823, the CPU 110 registers the previous imaging position and the latest imaging position as inappropriate positions. After step S823, the system 1 advances the process to step S812.

[0149] In step S812, the CPU 110 determines, based on the latest recognition result, one or more virtual viewpoints that increase the ease of recognition of one or more specific works 2S, as one or more imaging position candidates.

[0150] In step S824 after step S812, the CPU 110 records, for each of the one or more imaging position candidates, a model image ID that identifies the first model image 61 corresponding to each of the one or more specific workpieces 2S in the latest recognition result. At this time, the CPU 110 records the model image ID that identifies the first model image 61 in association with information on the imaging position candidate. The information on the imaging position candidate indicates, for example, the coordinates of the imaging position candidate.

[0151] In addition, when a plurality of imaging position candidates are determined in step S824, the CPU 110 records, for each imaging position candidate, a model image ID that identifies the first model image 61 corresponding to each specific workpiece 2S in association with the imaging position candidate information. In the subsequent step S9, at the stage where the next imaging position is determined from among the plurality of imaging position candidates, the model image ID that identifies the first model image 61 corresponding to the imaging position candidate other than the next imaging position is deleted from the recognition result.

[0152] <Second embodiment of the subroutine of step S8> The second embodiment of the subroutine of step S8 is a combination of the above-mentioned first embodiment and reference example 1. That is, in the second embodiment, one or more virtual viewpoints corresponding to one or more virtual depth maps having relatively large areas of missing regions are determined as one or more imaging position candidates, and then the imaging position candidates are narrowed down in consideration of the ease of recognition of the work 2.

[0153] Fig. 19 is a flow chart showing a second embodiment of the subroutine of step S8 shown in Fig. 8. As shown in Fig. 19, the second embodiment of the subroutine of step S8 includes steps S801 to S804, S811, and S812a.

[0154] As in the first embodiment of the subroutine of step S8, the CPU 110 performs the processes of steps S801 to S804. As a result, one or more virtual viewpoints corresponding to one or more virtual depth maps having relatively large areas of missing regions among the multiple virtual depth maps are temporarily determined as one or more imaging position candidates.

[0155] After step S804, the CPU 110 continues to perform the processes of steps S811 and S812a. Step S812a corresponds to step S812. In step S812a, the CPU 110 narrows down the one or more imaging position candidates determined in step S804 to imaging position candidates that increase the ease of recognition of one or more specific workpieces 2S based on the latest recognition result. Specifically, the CPU 110 narrows down the one or more imaging position candidates determined in step S804 to one or more imaging position candidates corresponding to one or more virtual viewpoints that satisfy a specific condition.

[0156] <Third Example of the Subroutine of Step S8> The third embodiment of the subroutine of step S8 is a combination of the first embodiment and the second reference example. That is, in the third embodiment, like the second embodiment, one or more virtual viewpoints corresponding to one or more virtual depth maps having relatively large areas of missing areas are determined as one or more imaging position candidates, and then the imaging position candidates are narrowed down in consideration of the ease of recognition of the work 2. Furthermore, the third embodiment registers inappropriate positions according to the recognition results.

[0157] Fig. 20 is a flow chart showing a third embodiment of the subroutine of step S8 shown in Fig. 8. As shown in Fig. 20, the third embodiment of the subroutine of step S8 includes steps S801 to S804, S812a, and S821 to S824.

[0158] As in the first embodiment of the subroutine of step S8, the CPU 110 performs the processes of steps S801 to S804. As a result, one or more virtual viewpoints corresponding to one or more virtual depth maps having relatively large areas of missing regions among the multiple virtual depth maps are temporarily determined as one or more imaging position candidates.

[0159] After step S804, CPU 110 performs the processes of steps S821, S822, S823, S812a, and S824. In step S823, an inappropriate position is registered, as in the second reference example.

[0160] Details of step S812a are as described in Example 2. As a result, the one or more imaging position candidates determined in step S804 are narrowed down to imaging position candidates that increase the ease of recognition of the one or more specific works 2S.

[0161] <First Example of Subroutine of Step S9> Fig. 21 is a flow chart showing a first embodiment of the subroutine of step S9 shown in Fig. 8. The first embodiment of the subroutine of step S9 includes steps S901 to S909.

[0162] First, in step S901, the CPU 110 operating as the priority setting unit 21 of the image processing device 100 sets a priority for each of the determined one or more imaging position candidates based on one or more indices. For example, for each imaging position candidate, the CPU 110 calculates at least one of the score Sa determined from the area of ​​the missing area in the corresponding virtual depth map, the score Sb determined from the movement amount of the robot arm 301 for moving the three-dimensional camera 200 to the imaging position candidate, and the score Sc determined from the recognition ease of the workpiece 2 when viewed from the imaging position candidate. The CPU 110 sets the priority based on the calculated score. When the CPU 110 calculates only one of the scores Sa, Sb, and Sc, the CPU 110 sets the priority in the order of the calculated score values. When the CPU 110 calculates two or more scores of the scores Sa, Sb, and Sc, the CPU 110 calculates an integrated score from the values ​​of the calculated two or more scores, and sets the priority in the order of the integrated score values.

[0163] In the next step S902, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 selects the imaging position candidate with the highest priority.

[0164] In the next step S903, the CPU 110 generates an inquiry as to whether or not the three-dimensional camera 200 can be moved to the selected imaging position candidate, and outputs the inquiry to the robot controller 400.

[0165] In the next step S904, the control processing circuit 420 operating as the movement possibility determination unit 42 of the robot controller 400 determines whether or not the three-dimensional camera 200 can be moved to the selected imaging position candidate.

[0166] If the three-dimensional camera 200 can be moved to the imaging position candidate (YES in step S904), in step S905 the control processing circuit 420 determines the selected imaging position candidate as the next imaging position. Furthermore, the control processing circuit 420 outputs to the image processing device 100 a determination result indicating that the three-dimensional camera 200 can be moved to the imaging position candidate and that the imaging position candidate will be the next imaging position. After step S905, the system 1 completes step S9.

[0167] If the three-dimensional camera 200 cannot be moved to the imaging position candidate (NO in step S904), in step S906 the control processing circuit 420 notifies the image processing device 100 of the determination result indicating that movement is not possible.

[0168] In step S907 after step S906, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines whether or not there is an unselected imaging position candidate.

[0169] If there is an unselected imaging position candidate (YES in step S907), the CPU 110 selects the imaging position candidate with the next highest priority in step S908. After step S908, the system 1 returns the process to step 903.

[0170] If there is no unselected imaging position candidate (NO in step S907), in step S909, the system 1 determines a position other than the one or more imaging position candidates as the next imaging position. After step S909, the system 1 completes step S9.

[0171] <First Example of Subroutine of Step S909> Fig. 22 is a flow chart showing a first embodiment of the subroutine of step S909 shown in Fig. 21. The first embodiment of the subroutine of step S909 includes step S911.

[0172] In step S911, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines the latest imaging position as the next imaging position. The CPU 110 outputs a movement command to the robot controller 400 to move the three-dimensional camera 200 to the next imaging position.

[0173] <Second embodiment of the subroutine of step S909> Fig. 23 is a flow chart showing a second embodiment of the subroutine of step S909 shown in Fig. 21. The second embodiment of the subroutine of step S909 includes steps S921 to S927 and S911.

[0174] First, in step S921, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 calculates the midpoint between the imaging position candidate with the highest priority and the latest imaging position.

[0175] In the next step S922, the CPU 110 generates an inquiry as to whether the three-dimensional camera 200 can be moved to the intermediate point, and outputs the inquiry to the robot controller 400.

[0176] In the next step S923, the control processing circuit 420 operating as the movement possibility determination unit 42 of the robot controller 400 determines whether or not the three-dimensional camera 200 can be moved to the midpoint.

[0177] If the three-dimensional camera 200 can be moved to the midpoint (YES in step S923), in step S924 the control processing circuit 420 determines the midpoint as the next imaging position. Furthermore, the control processing circuit 420 outputs a determination result indicating that the three-dimensional camera 200 can be moved to the midpoint and that the midpoint will be the next imaging position to the image processing device 100. After step S924, the system 1 completes step S9.

[0178] If the three-dimensional camera 200 cannot be moved to the midpoint (NO in step S923), then in step S925 the control processing circuit 420 outputs a determination result indicating that movement is not possible to the image processing device 100.

[0179] In step S926 after step S925, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines whether or not there is an imaging position candidate for which the midpoint with respect to the latest imaging position has not been calculated.

[0180] If there is an imaging position candidate for which the midpoint has not been calculated (YES in step S926), in step S927, the CPU 110 calculates the midpoint between the imaging position candidate with the next highest priority and the latest imaging position. After step S927, the system 1 returns the process to step 922.

[0181] If there is no imaging position candidate for which the midpoint has not been calculated (NO in step S926), in step S911, the CPU 110 determines the latest imaging position as the next imaging position and outputs a movement command to the robot controller 400 to move the three-dimensional camera 200 to the next imaging position.

[0182] <Second embodiment of the subroutine of step S9> Fig. 24 is a flow chart showing a second embodiment of the subroutine of step S9 shown in Fig. 8. The second embodiment of the subroutine of step S9 includes steps S901, and S931 to S936.

[0183] As in the first embodiment shown in FIG. 21, first, in step S901, the CPU 110 operating as the priority setting unit 21 of the image processing device 100 sets a priority for each of the one or more determined imaging position candidates based on one or more indicators.

[0184] In the next step S931, the CPU 110 generates an inquiry as to whether the three-dimensional camera 200 can move to all of the one or more imaging position candidates included in the list, and outputs the inquiry to the robot controller 400.

[0185] In the next step S932, the control processing circuit 420 operating as the movement possibility determination unit 42 of the robot controller 400 checks whether the three-dimensional camera 200 can be moved to each of the one or more imaging position candidates.

[0186] In the next step S933, the control processing circuit 420 determines whether or not there is a candidate imaging position to which the three-dimensional camera 200 can move.

[0187] If there is a candidate imaging position to which the three-dimensional camera 200 can move (YES in step S933), in step S934 the control processing circuit 420 determines the imaging position candidate with the highest priority among the one or more candidate imaging positions to which the three-dimensional camera 200 can move as the next imaging position. Furthermore, the control processing circuit 420 outputs a determination result indicating that there is a candidate imaging position to which the three-dimensional camera 200 can move and indicating the next imaging position to the image processing device 100. After step S934, the system 1 completes step S9.

[0188] If there are no candidate imaging positions to which the three-dimensional camera 200 can move (NO in step S933), in step S935, the control processing circuit 420 outputs a determination result indicating that there are no candidate imaging positions to which the three-dimensional camera 200 can move to the image processing device 100.

[0189] In step S936 after step S935, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines a position other than the one or more imaging position candidates as the next imaging position. After step S936, the system 1 completes step S9.

[0190] <First Example of Subroutine of Step S936> The first embodiment of the subroutine of step S936 includes step S911 shown in Fig. 22, similarly to the first embodiment of the subroutine of step S909. That is, in step S911, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines the latest imaging position as the next imaging position. The CPU 110 outputs a movement command to the robot controller 400 to move the 3D camera 200 to the next imaging position.

[0191] <Second embodiment of the subroutine of step S936> Fig. 25 is a flow chart showing a second embodiment of the subroutine of step S936 shown in Fig. 24. The second embodiment of the subroutine of step S936 includes steps S941 to S946 and S911.

[0192] First, in step S941, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 calculates the midpoint between each of one or more imaging position candidates included in the list and the latest imaging position.

[0193] In the next step S942, the CPU 110 generates an inquiry as to whether the three-dimensional camera 200 can be moved to each waypoint, and outputs the inquiry to the robot controller 400. Each waypoint is labeled with the priority of the corresponding imaging position candidate.

[0194] In the next step S943, the control processing circuit 420 operating as the movement feasibility determination unit 42 of the robot controller 400 checks whether the three-dimensional camera 200 can be moved to each of the intermediate points.

[0195] In the next step S944, the control processing circuit 420 determines whether or not there is an intermediate point to which the three-dimensional camera 200 can be moved.

[0196] If there is an intermediate point to which the three-dimensional camera 200 can move (YES in step S944), in step S945 the control processing circuit 420 determines the intermediate point corresponding to the imaging position candidate with the highest priority among the one or more intermediate points to which the three-dimensional camera 200 can move as the next imaging position. Furthermore, the control processing circuit 420 outputs a determination result indicating that there is a movable intermediate point and indicating the next imaging position to the image processing device 100. After step S945, the system 1 completes step S936.

[0197] If there is no intermediate point to which the three-dimensional camera 200 can move (NO in step S944), in step S946, the control processing circuit 420 outputs a determination result indicating that there is no intermediate point to which the three-dimensional camera 200 can move to the image processing device 100.

[0198] In step S911 after step S946, the CPU 110 operating as the candidate list management unit 23 of the image processing device 100 determines the latest imaging position as the next imaging position. The CPU 110 outputs a movement command to the robot controller 400 to move the three-dimensional camera 200 to the next imaging position.

[0199] <Modification> Fig. 26 is a flow chart showing an example of the flow of processing of a system according to a modified example. As shown in Fig. 26, the system 1 may execute step S7 and steps S8 and S9 in parallel. This allows the system 1 to move the three-dimensional camera 200 to the next imaging position more quickly than in the flow shown in Fig. 8 in which steps S7 to S9 are executed in order. As a result, the cycle time required to execute steps S3 to S10 is shortened.

[0200] In the above description, when the robot controller 400 determines in response to the inquiry that the three-dimensional camera 200 can be moved to an imaging position candidate, the robot controller 400 determines the imaging position candidate as the next imaging position. However, when the robot controller 400 determines that the three-dimensional camera 200 can be moved to an imaging position candidate, the robot controller 400 may notify the image processing device 100 of the determination result indicating that the movement is possible. The CPU 110 of the image processing device 100 may determine, as the next imaging position, one imaging position candidate to which it is determined that the three-dimensional camera 200 can be moved. After that, the CPU 110 may generate a movement command for moving the three-dimensional camera 200 to the next imaging position, and output the movement command to the robot controller 400.

[0201] The image processing device 100 may include a plurality of devices. Alternatively, the image processing device 100 may have a plurality of processors. In this case, the three-dimensional measurement unit 10, the viewpoint conversion unit 11, the missing area calculation unit 12, the determination unit 13, the model image generation unit 14, the assignment unit 15, the three-dimensional recognition unit 17, the picking position calculation unit 19, the range setting unit 20, the priority setting unit 21, the registration unit 22, and the candidate list management unit 23 may be realized by the same processor or by different processors.

[0202] In the above description, the robot controller 400 and the image processing device 100 are separate entities. However, the functions of the robot controller 400 may be incorporated into the image processing device 100.

[0203] §3 Supplementary Note As described above, the present embodiment includes the following disclosure.

[0204] (Configuration 1) An image processing device (100), a viewpoint conversion unit (11, 110) that converts a real depth map (50) obtained from an image captured by a three-dimensional camera (200) attached to a robot arm (301) into a plurality of virtual depth maps (52) viewed from a plurality of virtual viewpoints (VP); a calculation unit (12, 110) that calculates an area of ​​a missing region (56_1, 56_2) where distance information is missing for each of the plurality of virtual depth maps (52); and a determination unit (13, 110) that determines one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area among the plurality of virtual depth maps (52) as one or more next imaging position candidates for the three-dimensional camera (200).

[0205] (Configuration 2) a range setting unit (20, 110) that sets a movable range of the three-dimensional camera (200) in response to an input to a user interface (160); 2. The image processing device (100) according to configuration 1, wherein the viewpoint conversion unit (11, 110) arranges the multiple virtual viewpoints (VP) within the movable range.

[0206] (Configuration 3) The image processing device (100) according to configuration 1 or 2, wherein the missing regions (56_1, 56_2) do not include regions (54) outside a virtual border line (53) corresponding to an image frame of the real depth map (50) in each of the plurality of virtual depth maps (52).

[0207] (Configuration 4) The image processing device (100) according to any one of configurations 1 to 3, wherein the calculation unit (12, 110) determines, as the missing region (56_1, 56_2), a region in which pixels having missing distance information are connected by a predetermined number or more.

[0208] (Configuration 5) a priority setting unit (21, 110) that sets a priority for each of the one or more imaging position candidates based on one or more indexes including at least the area; an output unit (23, 110) that outputs an inquiry as to whether or not the three-dimensional camera (200) can be moved to an imaging position candidate selected from the one or more imaging position candidates to a robot controller (400) that controls the robot arm (301); The image processing device (100) according to any one of configurations 1 to 4, wherein the output unit (23, 110) selects an imaging position candidate to be queried from among the one or more imaging position candidates in descending order of priority.

[0209] (Configuration 6) The image processing device (100) according to configuration 5, wherein the one or more indicators include a movement amount of the robot arm (301) for moving the three-dimensional camera (200) to a target imaging position candidate among the one or more imaging position candidates.

[0210] (Configuration 7) 1. An image processing method, comprising: One or more processors (110) convert a real depth map (50) obtained from an image captured by a three-dimensional camera (200) attached to a robot arm (301) into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints (VP); The one or more processors (110) calculate an area of ​​a missing region where distance information is missing for each of the plurality of virtual depth maps (52); The image processing method includes the one or more processors (110) determining one or more virtual viewpoints corresponding to one or more virtual depth maps among the plurality of virtual depth maps (52) that have a relatively large area as one or more next imaging position candidates for the three-dimensional camera (200).

[0211] (Configuration 8) A program (115) for causing a computer to execute an image processing method, The image processing method includes: Converting a real depth map (50) obtained from an image captured by a three-dimensional camera (200) attached to a robot arm (301) into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints (VP); Calculating an area of ​​a missing region (56_1, 56_2) where distance information is missing for each of the plurality of virtual depth maps (52); and determining one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area among the plurality of virtual depth maps (52) as one or more next imaging position candidates for the three-dimensional camera (200).

[0212] Although the embodiment of the present invention has been described, the embodiment disclosed herein should be considered as illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0213] 1 system, 2 work, 2M work model, 2S specific work, 2a to 2c, 2z surface, 3 tray, 3a to 3c, 3z area, 10 three-dimensional measurement unit, 11 viewpoint conversion unit, 12 missing area calculation unit, 13 determination unit, 14 model image generation unit, 15 assignment unit, 16 model image set, 17 three-dimensional recognition unit, 18 recognition result storage unit, 19 picking position calculation unit, 20 range setting unit, 21 priority setting unit, 22 registration unit, 23 candidate list management unit, 24, 41 communication unit, 42 movement possibility determination unit, 43 control unit, 50 real depth map, 52 virtual depth map, 53 virtual frame line, 54 outside frame area, 56_1, 56_2 missing area, 60, ID model image, 61 first model image, 62 second model image, 100 image processing device, 106 Memory card, 110 CPU, 112,440 main memory, 114,450 storage, 115 program, 116 camera interface, 116a image buffer, 118 input interface, 120 display controller, 124 communication interface, 126 data reader / writer, 128 bus, 150 display device, 160 input device, 200 three-dimensional camera, 201 projection unit, 202 shooting unit, 250 optical axis, 300 robot, 301 robot arm, 302 base, 304 end effector, 400 robot controller, 410 network controller, 420 control processing circuit, 430 processor, 452 system program, 460 interface circuit, RP imaging position, VP virtual viewpoint.

Claims

1. An image processing device, a viewpoint conversion unit that converts a real depth map obtained from an image captured by a three-dimensional camera attached to the robot arm into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints; a calculation unit that calculates an area of ​​a missing region where distance information is missing for each of the plurality of virtual depth maps; a determination unit that determines one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area among the plurality of virtual depth maps as one or more next imaging position candidates for the three-dimensional camera.

2. a range setting unit that sets a movable range of the three-dimensional camera in response to an input to a user interface; The image processing device according to claim 1 , wherein the viewpoint conversion section arranges the plurality of virtual viewpoints within the movable range.

3. The image processing device according to claim 1 , wherein the missing region does not include a region outside a virtual frame line corresponding to an image frame of the real depth map in each of the plurality of virtual depth maps.

4. The image processing device according to claim 1 , wherein the calculation unit determines, as the missing region, a region in which a predetermined number or more of pixels in which the distance information is missing are connected.

5. a priority setting unit that sets a priority for each of the one or more imaging position candidates based on one or more indexes including at least the area; an output unit that outputs an inquiry as to whether or not the three-dimensional camera can be moved to an imaging position candidate selected from the one or more imaging position candidates to a robot controller that controls the robot arm, The image processing device according to claim 1 , wherein the output unit selects an imaging position candidate to be the subject of the inquiry from among the one or more imaging position candidates in descending order of priority.

6. The image processing apparatus according to claim 5 , wherein the one or more indicators include a movement amount of the robot arm for moving the three-dimensional camera to a target imaging position candidate among the one or more imaging position candidates.

7. 1. An image processing method, comprising: one or more processors converting a real depth map obtained from an image captured by a 3D camera attached to the robot arm into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints; The one or more processors calculate an area of ​​a missing region where distance information is missing for each of the plurality of virtual depth maps; The image processing method includes the one or more processors determining one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area among the plurality of virtual depth maps as one or more next candidate imaging positions for the three-dimensional camera.

8. A program for causing a computer to execute an image processing method, The image processing method includes: Converting a real depth map obtained from an image captured by a three-dimensional camera attached to a robot arm into a plurality of virtual depth maps viewed from a plurality of virtual viewpoints; Calculating an area of ​​a missing region where distance information is missing for each of the plurality of virtual depth maps; determining one or more virtual viewpoints corresponding to one or more virtual depth maps having a relatively large area among the plurality of virtual depth maps as one or more next imaging position candidates for the three-dimensional camera.

Images