Image processing device, image processing method and program

Abstract

To provide an image processing device capable of suppressing an increase in time required for object recognition processing.SOLUTION: An image processing device is provided with an image processing unit for repeatedly performing search processing for searching for candidates for an area in which an object exists in an input image by template matching using template images sequentially selected from among a plurality of template images showing the object or a model corresponding to the model. The image processing unit ends the search processing when the number of candidates reaches an upper limit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Conventionally, in production sites, robots are used that pick up workpieces one by one from among multiple workpieces that are piled up randomly. In order for the robot to pick up the workpiece to be picked up, the position and orientation of the workpiece must be identified. For this reason, an image processing device has been developed that performs object recognition processing to recognize the position and orientation of the workpiece to be picked up using an input image obtained by imaging multiple workpieces that are piled up randomly. The object recognition processing generally uses template matching using a template image in which a workpiece or a workpiece model is shown. Template matching is a technology that searches for an area similar to the template image in an input image. The workpieces that are piled up randomly can take various orientations. For this reason, multiple template images are prepared in advance according to the orientations that the workpieces can take.

[0003] The object recognition process includes a process of searching for multiple candidates for the area where the workpiece exists in the input image by template matching, and a process of narrowing down the multiple candidates to candidates with relatively high matching scores (hereinafter simply referred to as "scores"). The score indicates the degree of match with the template image.

[0004] When the number of workpieces in the input image is large, the number of candidates to be searched also increases. As a result, the time required for object recognition processing increases. If the time required for object recognition processing increases, the start of picking by the robot is delayed, and production efficiency decreases. Therefore, it is desirable to shorten the time required for object recognition processing.

[0005] Japanese Patent Laid-Open Publication No. 2016-148558 (Patent Document 1) discloses an information processing device that identifies some small areas in the existence range of a group of objects in ascending order of distance value based on the result of distance measurement for the group of objects, and determines an area that includes the small areas and is larger than the small areas as a search area for objects. This allows objects that are located high up in a pile of objects to be quickly identified. Japanese Patent Laid-Open Publication No. 2018-146347 (Patent Document 2) and Japanese Patent Laid-Open Publication No. 2017-170567 (Patent Document 3) also disclose a technology for limiting a search area based on the height of a workpiece. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-148558 A [Patent Document 2] JP 2018-146347 A [Patent Document 3] JP 2017-170567 A Summary of the Invention [Problem to be solved by the invention]

[0007] In the techniques disclosed in Patent Documents 1 to 3, the search area depends on the height of the workpiece. Therefore, the time required for the object recognition process may depend on the state of the workpieces being stacked. On the other hand, in order to prevent a delay in starting picking, it is desirable to limit the time required for the object recognition process to a certain range regardless of the state of the workpieces being stacked.

[0008] The present disclosure has been made in consideration of the above circumstances, and has as its objective to provide an image processing device, an image processing method, and a program that are capable of suppressing an increase in the time required for object recognition processing. [Means for solving the problem]

[0009] According to an example of the present disclosure, an image processing device includes an image processing unit that repeatedly performs a search process to search for candidates for an area where an object exists in an input image by template matching using template images selected in order based on image features of the template images from among a plurality of template images depicting an object or a model corresponding to the object. The image processing unit ends the search process when the number of candidates reaches an upper limit.

[0010] According to this disclosure, even if the number of objects captured in an input image is large, the number of candidates is limited to an upper limit, thereby suppressing an increase in the time required for object recognition processing.

[0011] In the above disclosure, each of the multiple template images is classified into one of multiple image groups having different viewpoint directions with respect to an object or model, and the image processing unit preferentially selects, from the multiple template images, a template image belonging to an image group having a relatively large area of ​​an orthogonal projection of the object or model along the viewpoint direction.

[0012] According to this disclosure, a template image with a relatively low degree of freedom for matching is preferentially selected, thereby reducing the frequency of extracting erroneously searched regions as candidates.

[0013] In the above disclosure, two or more template images belonging to each of a plurality of image groups are classified into one of a plurality of image subgroups having mutually different distances between the viewpoint and the object or model, and the image processing unit preferentially selects, from among the two or more template images, a template image belonging to an image subgroup having a relatively short distance.

[0014] Alternatively, each of the plurality of template images is classified into one of a plurality of image groups having different distances between the object or model and the viewpoint, and the image processing unit preferentially selects, from the plurality of template images, a template image belonging to an image group having a relatively short distance.

[0015] According to these disclosures, an area in which an object located closer to a camera capturing an input image exists is more likely to be extracted as a candidate. Therefore, when multiple objects are piled up randomly, an area in which an object located higher exists is more likely to be extracted as a candidate. When picking one of multiple objects piled up randomly, an object located higher is generally easier to pick. This makes it easier to recognize the position and orientation of an object that is easy to pick.

[0016] In the above disclosure, the image processing unit performs template matching with priority on a specific range of the input image, which makes it easier to extract candidates from within the specific range.

[0017] In the above disclosure, the upper limit value is set according to a user input. According to this disclosure, the user can adjust the time required for object recognition processing by appropriately changing the upper limit value.

[0018] In the above disclosure, the image processing device further includes a setting unit that sets an upper limit value. The setting unit sets the upper limit value based on the result of the search process using the sample image and the target processing time. According to this disclosure, the upper limit value according to the target processing time is automatically set.

[0019] In the above disclosure, the image processing unit further performs at least one post-stage process following the search process. The setting unit calculates a first time required to search for one candidate from a result of the search process using the sample image. The setting unit calculates a second time required for the post-stage process for each candidate and a decay rate of the number of candidates due to the post-stage process from each result of the at least one post-stage process using the sample image. The setting unit calculates an upper limit value using the first time, the second time, the decay rate, and a target processing time.

[0020] According to this disclosure, an appropriate upper limit value according to the target processing time is automatically set, taking into consideration at least one subsequent process following the search process.

[0021] According to another example of the present disclosure, an image processing method includes a processor selecting one template image from a plurality of template images depicting an object or a model corresponding to the object based on image features of the template image, and a processor searching for candidates for an area in which the object exists from an input image by template matching using the one template image. The selection of the one template image and the search for the candidates are repeated until the number of candidates reaches an upper limit. The image processing method further includes the processor terminating the selection of the one template image and the search for the candidates in response to the number of candidates reaching the upper limit.

[0022] According to yet another example of the present disclosure, a program causes a computer to execute the image processing method described above. According to these disclosures, the image processing method or program can also suppress an increase in the time required for object recognition processing. Effect of the Invention

[0023] According to the present disclosure, an image processing device, an image processing method, and a program can suppress an increase in the time required for object recognition processing. [Brief description of the drawings]

[0024] [Figure 1] 1 is a schematic diagram showing an overall configuration of a control system according to an embodiment; [Diagram 2] 2 is a schematic diagram illustrating an example of a hardware configuration of the image processing device illustrated in FIG. 1. [Diagram 3] 2 is a block diagram showing an example of a functional configuration of the image processing device shown in FIG. 1. [Figure 4] 13 is a flowchart showing an example of the flow of a template image generation process. [Diagram 5] 11A to 11C are diagrams illustrating a generation process of a template image. [Figure 6] 11 is a flowchart showing an example of the flow of an object recognition process. [Figure 7]FIG. 2 illustrates images used in pyramid searching. [Figure 8] 7 is a flowchart showing the process flow of a subroutine of step S12 shown in FIG. 6. [Figure 9] 9 is a flowchart showing the process flow of a subroutine of step S20 shown in FIG. 8. [Figure 10] 9 is a flowchart showing the process flow of a subroutine of step S22 shown in FIG. 8. [Figure 11] FIG. 11 is a diagram illustrating an example of a method for setting an upper limit value. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and the description thereof will not be repeated.

[0026] §1 Examples of application An example of a situation in which the present invention is applied will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an overall configuration of a control system according to an embodiment. The control system 1 illustrated in Fig. 1 is incorporated in a production line or the like, and controls pick-and-place of workpieces 2 randomly piled in a container 3. The position and orientation of the workpieces 2 within the container 3 are random. The workpieces 2 are an example of an "object" in this disclosure.

[0027] As shown in FIG. 1, the control system 1 includes an image processing device 100, a robot controller 200, a robot 300, and a measuring head 400.

[0028] A display device 150 and an input device 160 are connected to the image processing device 100. The display device 150 is typically a liquid crystal display and displays various screens. The input device 160 includes, for example, a keyboard and a mouse. The display device 150 and the input device 160 may be integrated to form a touch panel.

[0029] The robot 300 performs pick-and-place for the workpiece 2. The robot 300 includes an end effector 301 that holds the workpiece 2, an articulated arm 302 for changing the position and posture of the end effector 301, and a base 303 that supports the articulated arm 302. The end effector 301 is attached to the surface (flange surface) of the tip of the articulated arm 302. The operation of the robot 300 is controlled by the robot controller 200.

[0030] The robot controller 200 receives information from the image processing device 100 and controls the articulated arm 302 of the robot 300. Specifically, the robot controller 200 controls the articulated arm 302 so that the end effector 301 performs an approach operation, a picking operation, and a placing operation in that order. The "approach operation" is a movement operation from outside the container 3 to a position and posture where the end effector 301 can grasp the workpiece 2. The "picking operation" is a movement operation in which the end effector 301 grasps the workpiece 2. The "place operation" is a movement operation in which the end effector 301 grasps the workpiece 2 and transports it to the target position while grasping the workpiece 2.

[0031] The robot controller 200 receives from the image processing device 100 position and orientation information indicating the position and orientation to be taken by the end effector 301 at the start of a picking operation. Based on the position and orientation information, the robot controller 200 generates commands to the robot 300 so that the end effector 301 sequentially executes an approach operation, a picking operation, and a placing operation. In response to the commands, the end effector 301 of the robot 300 sequentially executes an approach operation, a picking operation, and a placing operation.

[0032] The measurement head 400 is installed so that the workpiece 2 accommodated in the container 3 becomes a subject, and captures an image of the subject. The measurement head 400 includes a projection unit 401 and an image capture unit 402. The projection unit 401 projects an arbitrary projection pattern light onto the subject in accordance with an instruction from the image processing device 100. The projection pattern is, for example, a pattern in which brightness changes periodically along a predetermined direction within an irradiation surface. The image capture unit 402 captures an image of the subject with the projection pattern light projected onto it.

[0033] The projection unit 401 includes, as main components, a light source such as an LED (Light Emitting Diode) or a halogen lamp, and a filter arranged on the side of the irradiation surface of the projection unit 401. The filter generates a projection pattern light required for measuring a three-dimensional shape as described later, and can arbitrarily change the light transmittance in the surface according to a command from the image processing device 100. The projection unit 401 may project a fixed projection pattern light that does not change over time, or may project a projection pattern light that changes over time. The projection unit 401 may be configured using a liquid crystal or a DMD (Digital Mirror Device) and a light source (such as an LED or laser light source).

[0034] The photographing unit 402 includes, as main components, an optical system such as a lens, and an image pickup element such as a CCD (Coupled Charged Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0035] The image processing device 100 includes an image processing unit 10 as a main component. The image processing unit 10 performs object recognition processing to recognize the position and orientation of the workpiece 2 to be picked, using an input image 20 that represents the three-dimensional shape of the field of view of the measurement head 400. The input image 20 is, for example, a distance image, and is generated based on a captured image received from the measurement head 400. The image processing device 100 determines the position and orientation of the end effector 301 when picking up the workpiece 2 to be picked, based on the position and orientation recognized by the object recognition processing. The image processing device 100 generates position and orientation information indicating the determined position and orientation, and outputs the generated position and orientation information to the robot controller 200.

[0036] As part of the object recognition process, the image processing unit 10 repeatedly performs a search process by template matching to search for candidates for an area in which the workpiece 2 exists within the input image 20. The template matching uses template images selected in order based on the image features of the template images 30 from among a plurality of template images 30 in which the workpiece 2 or a workpiece model corresponding to the workpiece 2 is depicted. However, the image processing unit 10 ends the search process in response to the number of candidates reaching an upper limit value. For example, when the image processing unit 10 searches for the same number of candidates as the upper limit value by template matching using the template images 30_1 to 30_k, it terminates the template matching using the remaining template images 30 (template images from template image 30_k+1 onward).

[0037] According to this embodiment, even if the number of workpieces 2 appearing in the input image 20 is large, the number of candidates is limited to an upper limit value. Therefore, an increase in the time required for object recognition processing is suppressed.

[0038] §2 Specific examples <Example of hardware configuration of image processing device> The image processing device 100 is typically a computer having a general-purpose architecture, and executes a pre-installed program (instruction code) to perform the image processing according to the present embodiment. Such a program is typically distributed in a state stored in various recording media, or is installed in the image processing device 100 via a network, etc.

[0039] When such a general-purpose computer is used, in addition to the application for executing the image processing according to the present embodiment, an OS (Operating System) for executing basic processing of the computer may be installed. In this case, the program according to the present embodiment may call necessary modules among program modules provided as part of the OS in a predetermined sequence at a predetermined timing to execute processing. That is, the program according to the present embodiment itself may not include the above-mentioned modules, and may execute processing in cooperation with the OS. The program according to the present embodiment may be in a form that does not include some of these modules.

[0040] Furthermore, the program according to the present embodiment may be provided by being incorporated into a part of another program. In this case, the program itself does not include the modules included in the other program to be combined as described above, and the processing is executed in cooperation with the other program. That is, the program according to the present embodiment may be in a form incorporated into such other program. Note that some or all of the functions provided by the execution of the program may be implemented as a dedicated hardware circuit.

[0041] Fig. 2 is a schematic diagram showing an example of a hardware configuration of the image processing device shown in Fig. 1. As shown in Fig. 2, the image processing device 100 includes a CPU (Central Processing Unit) 110, which is an example of a processor, a main memory 112, a hard disk 114, a measuring head interface 116, an input interface 118, a display controller 120, a communication interface 124, and a data reader / writer 126. These components are connected to each other via a bus 128 so as to be able to communicate data with each other.

[0042] The CPU 110 loads the programs 115 installed on the hard disk 114 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a dynamic random access memory (DRAM), and in addition to the programs 115 read from the hard disk 114, it holds images acquired by the measurement head 400, etc. Furthermore, the hard disk 114 stores various data, etc., as described below. In addition to the hard disk 114, or instead of the hard disk 114, a semiconductor storage device such as a flash memory may be adopted.

[0043] The measurement head interface 116 mediates data transmission between the CPU 110 and the measurement head 400. That is, the measurement head interface 116 is connected to the measurement head 400. The measurement head interface 116 provides projection commands and image capture commands to the measurement head 400 in accordance with internal commands generated by the CPU 110. The measurement head interface 116 includes an image buffer 116a for temporarily storing images from the measurement head 400. When a predetermined number of images are stored in the image buffer 116a, the measurement head interface 116 transfers the stored images to the main memory 112.

[0044] The input interface 118 mediates data transmission between the CPU 110 and the input device 160. That is, the input interface 118 accepts input information input to the input device 160 by the user.

[0045] The display controller 120 is connected to a display device 150, and controls the screen of the display device 150 so as to notify the user of the processing results of the CPU 110, etc.

[0046] The communication interface 124 mediates data transmission between the CPU 110 and external devices such as the robot controller 200. The communication interface 124 is typically implemented by Ethernet (registered trademark) or USB (Universal Serial Bus).

[0047] The data reader / writer 126 mediates data transmission between the CPU 110 and the memory card 106, which is a recording medium. That is, the memory card 106 is distributed in a state in which a program executed by the image processing device 100 and the like are stored, and the data reader / writer 126 reads the program from the memory card 106. In addition, the data reader / writer 126 writes an image captured by the measurement head 400 and / or a processing result in the image processing device 100, and the like, to the memory card 106 in response to an internal command from the CPU 110. The memory card 106 is formed of a general-purpose semiconductor storage device such as an SD (Secure Digital), a magnetic storage medium such as a flexible disk, an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory), and the like.

[0048] <Example of functional configuration of image processing device> Fig. 3 is a block diagram showing an example of a functional configuration of the image processing device shown in Fig. 1. As shown in Fig. 3, the image processing device 100 includes an image processing unit 10, a template image generating unit 11, a head control unit 12, an input image generating unit 13, a setting unit 14, and a storage unit 15.

[0049] The image processing unit 10, the template image generating unit 11, the head control unit 12, the input image generating unit 13, and the setting unit 14 are realized by the CPU 110 shown in Fig. 2 executing a program 115. The storage unit 15 is realized by the main memory 112 or the hard disk 114.

[0050] The storage unit 15 prestores CAD (Computer Aided Design) data 16 indicating the three-dimensional shape of the workpiece 2. Furthermore, the storage unit 15 stores a plurality of template images 30 generated by the template image generation unit 11, a captured image 17 obtained by capturing an image of the measurement head 400, and an input image 20 generated by the input image generation unit 13.

[0051] The template image generating unit 11 generates a plurality of template images 30 in which a workpiece model corresponding to the workpiece 2 is depicted, based on the CAD data 16. The template image generating unit 11 stores the plurality of template images 30 in the storage unit 15.

[0052] The head control unit 12 controls the operation of the measurement head 400. The head control unit 12 outputs a projection command to instruct the measurement head 400 to project a predetermined projection pattern light. The head control unit 12 outputs an image capture command to the measurement head 400 while the projection pattern light is being projected. The head control unit 12 stores the captured image 17 acquired from the measurement head 400 in the storage unit 15.

[0053] The input image generating unit 13 performs three-dimensional measurement in the field of view of the measurement head 400 based on the captured image 17, and generates a distance image representing the three-dimensional shape of the field of view as the input image 20. For example, the input image generating unit 13 performs three-dimensional measurement processing using a triangulation method. As the triangulation method, known techniques such as a fixed pattern projection method, a phase shift method, a stereo matching method, and a light section method can be adopted. For example, the phase shift method is a method that uses an image captured in a state where a projection pattern light in which the shading is changed sinusoidally is projected onto a subject. A plurality of projection patterns with different shading change periods and phases are prepared, and the projection unit 401 of the measurement head 400 sequentially irradiates the plurality of projection pattern lights onto the field of view of the imaging unit 402. The input image generating unit 13 measures the three-dimensional shape of the field of view based on the change in brightness (luminance and brightness) in the captured image 17 when the plurality of projection patterns are projected. The three-dimensional measurement process is not limited to the triangulation method, and may be performed using other methods (for example, a coaxial method, etc.).

[0054] The setting unit 14 sets an upper limit value for the number of candidates. For example, the setting unit 14 sets the upper limit value according to a user input. That is, the setting unit 14 sets the value input by the user as the upper limit value. Alternatively, the setting unit 14 may set the upper limit value based on the result of the three-dimensional object recognition processing using the sample image so that the time for the three-dimensional object recognition processing is within a target processing time.

[0055] The image processing unit 10 performs object recognition processing to recognize the position and orientation of the workpiece 2 to be picked up.

[0056] Furthermore, the image processing unit 10 may perform interference determination processing based on the position and orientation of the workpiece 2 to be picked. The interference determination processing is processing for determining whether or not the robot 300 will interfere with an object (including a container 3 or another workpiece 2) other than the workpiece 2 to be picked when the end effector 301 grips the workpiece 2 to be picked. The relative position and orientation of the end effector 301 gripping the workpiece 2 with respect to the workpiece 2 is set in advance. The interference determination processing calculates a gripping position of the end effector 301 at which the robot 300 can grip the workpiece 2 to be picked without interfering with an object other than the workpiece 2 to be picked.

[0057] In the interference detection process, it is mainly confirmed whether the following conditions (a) to (c) are satisfied. (a) The end effector 301 does not interfere with the container 3, the floor surface or other workpieces 2. (b) The approach angle of the end effector 301 is within the reference range. (c) The rotation amount of the end effector 301 is within a reference range. The image processing unit 10 calculates a gripping position that satisfies the conditions (a) to (c). When there are multiple gripping positions that satisfy the conditions (a) to (c), the image processing unit 10 may preferentially select the gripping position that is closest to the measurement head 400. When the positions and orientations of multiple workpieces 2 to be picked are recognized by the object recognition process, the image processing unit 10 may preferentially select the gripping position corresponding to the workpiece 2 with the position and orientation having the highest score.

[0058] <Template image generation process> The template image generation process will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing an example of the flow of the template image generation process. Fig. 5 is a diagram for explaining the template image generation process.

[0059] First, in step S1, the CPU 110 operating as the template image generating unit 11 acquires CAD data 16. The CPU 110 generates a work model 2M (see FIG. 5) having a three-dimensional shape represented by the CAD data 16, and places the work model 2M at the origin in the virtual space.

[0060] In the next step S2, the CPU 110 sets a plurality of half lines 31 (see FIG. 5) having the origin as one end in the virtual space. The CPU 110 may set the plurality of half lines 31 in response to a user input. Alternatively, the CPU 110 may randomly set the plurality of half lines 31 in the virtual space such that the angle between two adjacent half lines 31 is constant.

[0061] Next, the CPU 110 performs a loop process L1 for the ray 31 selected in order from among the plurality of ray 31. The loop process L1 includes a loop process L2 for each of a plurality of predetermined distances D (distances D1 to Dm).

[0062] Loop process L2 includes steps S3 and S4 and loop process L3. In step S3, CPU 110 places viewpoint 32 at a position on half line 31 that is a distance D away from the origin. That is, distance D represents the distance between workpiece model 2M and viewpoint 32. Since viewpoint 32 is placed on half line 31, half line 31 represents the direction of viewpoint 32 relative to workpiece model 2M.

[0063] In the next step S4, the CPU 110 generates a virtual distance image of the workpiece model 2M when viewed from the viewpoint 32 as a template image 30. The resolution of the template image 30 is the same as the resolution of the input image 20. However, it is sufficient that the template image 30 has a size slightly larger than the area in which the workpiece model 2M appears. Therefore, the size (number of pixels) of the template image 30 is smaller than the size (number of pixels) of the input image 20.

[0064] The loop process L3 is executed for each of a plurality of predetermined rotation angles θ. The loop process L3 includes a step S5. In step S5, the CPU 110 generates a new template image 30 by rotating the template image 30 generated in step S4 by the corresponding rotation angle θ.

[0065] When the loop process L3 for all of the rotation angles θ is completed, the loop process L2 for one distance D is completed. When the loop process L2 for all of the distances D is completed, the loop process L1 for one ray 31 is completed. When the loop process L1 for all the ray 31 is completed, the template image generation process is completed.

[0066] A plurality of template images 30 are generated by the template image generation process according to the flow shown in FIG.

[0067] Note that the size of the work model 2M is different in multiple template images 30 that correspond to a certain half line 31 and different distances D. Therefore, the CPU 110 may execute steps S3 and S4 only for a predetermined reference distance D to generate a template image 30 (hereinafter referred to as a "reference template image"), and may generate template images 30 corresponding to other distances D by enlarging or reducing the reference template image.

[0068] 5, each of the multiple template images 30 is classified into one of multiple image groups 34 having different directions of the viewpoint 32 with respect to the work model 2M. For example, the template image 30 generated by the loop process L1 for the ray 31a is classified into the image group 34a corresponding to the direction of the viewpoint 32 represented by the ray 31a.

[0069] Furthermore, the two or more template images 30 classified into each of the multiple image groups 34 are classified into one of multiple image subgroups 35 having mutually different distances between the viewpoint 32 and the work model 2M. For example, among the two or more template images 30 classified into the image group 34a, the template image 30 generated by the loop process L2 for the distance D1 is classified into the image subgroup 35a.

[0070] <Object recognition processing> The object recognition process will be described with reference to Fig. 6 to Fig. 10. Fig. 6 is a flowchart showing an example of the flow of the object recognition process.

[0071] First, in step S11, the CPU 110 operating as the image processing unit 10 converts the input image 20 into a first feature image according to a predetermined algorithm. Furthermore, the CPU 110 converts each of the multiple template images 30 into a second feature image according to the same algorithm. Each pixel of the first feature image and the second feature image indicates a feature of the three-dimensional shape of the subject. The feature represents, for example, the degree of the edge of the subject or the normal direction of the surface of the subject. Note that the second feature image corresponding to each of the multiple template images 30 may be created in advance before the object recognition process.

[0072] In the next step S12, the CPU 110 performs a pyramid search using template matching. The pyramid search is also called a coarse-to-fine search.

[0073] FIG. 7 is a diagram showing images used in pyramid search. In pyramid search, a group of n images (image pyramid) from the first layer to the nth layer (n is an integer of 2 or more) with stepwise different resolutions is used. The resolution of the first layer image is the lowest, and the resolution increases in the order of the second layer, ..., and the nth layer. Compared with the i-th layer image (i=1, 2, ..., n-1), the (i+1)th layer image has a resolution p times larger in the horizontal direction and q times larger in the vertical direction (p, q are integers of 2 or more). The vertical and horizontal magnifications may be different (i.e., p ≠ q). When p = 2 and q = 2, the resolution decreases by 1 / 2 for each layer from the nth layer image to the first layer image. The CPU 110 generates first to nth layer images for the first feature image generated from the input image 20, and also generates first to nth layer images for the second feature image generated from each template image 30. Hereinafter, the first to n-th layer images generated from the first feature image are referred to as "first to n-th layer images (1)." The first to n-th layer images generated from the second feature image are referred to as "first to n-th layer images (2)."

[0074] FIG. 8 is a flowchart showing the flow of the process of the subroutine of step S12 shown in FIG. 6. In step S20, CPU 110 performs a search process using template matching on the first layer image (1) having the lowest resolution. In the next step S21, CPU 110 sets k=2. In the next step S22, CPU 110 performs a search process using template matching on the k-th layer image (1). In the next step S23, CPU 110 judges whether or not k=n is satisfied. If k=n is not satisfied (NO in step S23), in step S24, CPU 110 adds 1 to k. After step S24, the process returns to step S22. If k=n is satisfied (YES in step S23), the process of step S12 ends.

[0075] Fig. 9 is a flowchart showing the flow of the process of the subroutine of step S20 shown in Fig. 8. First, the CPU 110 performs loop process L4 on template images 30 selected in order from among the multiple template images 30 based on the image features of the template images 30. The loop process L4 includes loop process L5 for each pixel (x, y) in the first layer image (1) corresponding to the input image 20. The loop process L5 includes steps S30 to S33. The loop process L5 for pixel (xi, yj) will be described below.

[0076] In step S30, the CPU 110 calculates a score indicating the degree of match between the attention area corresponding to pixel (xi, yj) in the first layer image (1) and the first layer image (2) corresponding to the selected template image 30. The attention area has the same size as the first layer image (2) and is centered on pixel (xi, yj).

[0077] In step S31, the CPU 110 determines whether the score is equal to or greater than the threshold. If the score is equal to or greater than the threshold (YES in step S31), in step S32, the CPU 110 determines the attention area corresponding to pixel (xi, yj) as a candidate for the area in which the workpiece 2 exists. The CPU 110 records candidate information indicating the candidate. The candidate information includes the coordinates of pixel (xi, yj) and information identifying the selected template image 30.

[0078] In step S33 after step S32, CPU 110 determines whether or not the number of candidates has reached the upper limit value. If the number of candidates has reached the upper limit value (YES in step S33), step S20 ends.

[0079] If the score is less than the threshold value (NO in step S31), or if the number of candidates has not reached the upper limit (NO in step S33), the loop process L5 for pixel (xi, yj) is completed.

[0080] According to the flow shown in FIG. 9, even if the number of workpieces 2 shown in the input image 20 is large, the search process for the first layer image (1) ends when the number of candidates reaches an upper limit value.

[0081] FIG. 10 is a flowchart showing the flow of the process of the subroutine of step S22 shown in FIG. 8. First, CPU 110 performs loop process L6 for each of the candidates searched for in the same number as the upper limit value (or a number less than the upper limit value) in step S20. Loop process L6 includes step S40 and loop process L7. In step S40, CPU 110 sets a range including the position indicated by the corresponding candidate information and its periphery as a search range. After step S40, CPU 110 performs loop process L7. Loop process L7 is repeatedly performed for each pixel (x, y) in the search range set in step S40 in the k-th layer image (1) corresponding to input image 20. Loop process L7 includes steps S41 to S44. Hereinafter, loop process L7 for pixel (xi, yj) will be described.

[0082] In step S41, the CPU 110 calculates a score indicating the degree of match between the attention area corresponding to pixel (xi, yj) in the kth layer image (1) and the kth layer image (2) corresponding to the template image 30 indicated by the corresponding candidate information. The attention area has the same size as the kth layer image (2) and is centered on pixel (xi, yj).

[0083] In step S42, the CPU 110 determines whether the score is equal to or greater than the threshold. If the score is equal to or greater than the threshold (YES in step S42), in step S43, the CPU 110 leaves the attention area corresponding to pixel (xi, yj) as a candidate for the area in which the work 2 exists. The CPU 110 records candidate information indicating the candidate. The candidate information indicates pixel (xi, yj) and the template image 30. If the score is less than the threshold (NO in step S42), in step S44, the CPU 110 excludes pixel (xi, yj) from the candidates. After steps S43 and S44, the loop process L7 for pixel (xi, yj) ends.

[0084] When the loop process L7 is completed for all pixels in the search range, the loop process L6 for one candidate is completed. When the loop process L6 is completed for all candidates, step S22 ends.

[0085] As described above, even if the number of workpieces 2 in the input image 20 is large, the search process for the first layer image (1) ends when the number of candidates reaches the upper limit. Therefore, in step S22, the loop process L6 is repeated a maximum number of times equal to the upper limit. Therefore, the processing time of step S22 is prevented from becoming long.

[0086] 6, in step S13 after step S12, CPU 110 narrows down the candidates. Specifically, CPU 110 performs deletion of similar candidates, adjustment of candidate positions, sub-pixel position estimation, etc. In step S13, the multiple candidates are narrowed down to a predetermined number of candidates.

[0087] In the next step S14, the CPU 110 performs detailed alignment processing for each of the predetermined number of candidates. The detailed alignment processing includes processing for generating model point cloud data indicating the three-dimensional shape of the workpiece model 2M corresponding to the candidate, and processing for correcting the model point cloud data so as to minimize deviation from the point cloud data corresponding to the input image 20. The corrected model point cloud data is determined as data representing the position and orientation of the workpiece 2 in the three-dimensional space.

[0088] In the next step S15, the CPU 110 performs a final narrowing process. The final narrowing process includes a process of evaluating the model point cloud data identified in step S18, a process of selecting one or several final candidates from a predetermined number of candidates based on the evaluation result, and a process of outputting the position and orientation indicated by the model point cloud data corresponding to the selected final candidate. After step S15, the object recognition process ends.

[0089] <Template image priority> 9, loop processing L4 is performed on template images 30 selected in order from among the multiple template images 30 based on the image features of the template images 30. It is preferable that the CPU 110 preferentially selects, from among the multiple template images 30, template images 30 that belong to an image group 34 in which the area of ​​the orthogonal projection of the work model 2M along the direction of the half line 31 (the direction of the viewpoint 32 relative to the work model 2M) is relatively large.

[0090] When the workpiece 2 is placed on the floor surface, the larger the area of ​​the orthogonal projection of the workpiece 2 when viewed vertically, the more stable the posture of the workpiece 2. As shown in Fig. 1, the measurement head 400 is usually placed above the container 3. Therefore, by preferentially selecting a template image 30 that belongs to an image group 34 in which the area of ​​the orthogonal projection of the workpiece model 2M is relatively large, it becomes easier to search for a candidate area where a workpiece 2 with a stable posture exists.

[0091] In addition, by preferentially selecting template images 30 belonging to an image group 34 in which the area of ​​the orthogonal projection of the work model 2M is relatively large, it becomes easier to search for candidate areas in which the work 2 exists where more gripping positions by the end effector 301 can be calculated.

[0092] The degree of freedom in matching varies depending on the template image 30. When template matching is performed using a template image 30 with a high degree of freedom in matching, the possibility of erroneous detection increases. Therefore, when the number of candidates reaches an upper limit value by performing loop process L4 for a template image 30 with a relatively high degree of freedom in matching, the frequency of extracting an erroneously searched area as a candidate increases. The smaller the area of ​​the orthogonal projection of the work model 2M of a template image 30 is, the higher the degree of freedom in matching becomes. By preferentially selecting a template image 30 belonging to an image group 34 with a relatively large area of ​​the orthogonal projection of the work model 2M, the frequency of extracting an erroneously searched area as a candidate decreases.

[0093] Furthermore, the robot 300 can easily pick up a workpiece 2 located at a relatively high position among the multiple workpieces 2 piled up randomly. The higher the workpiece 2 located, the shorter the distance from the measurement head 400. Therefore, it is preferable that the CPU 110 preferentially selects, from among two or more template images 30 belonging to a certain image group 34, a template image 30 belonging to an image subgroup 35 in which the distance D between the workpiece model 2M and the viewpoint 32 is relatively short.

[0094] Note that the multiple template images 30 belonging to the image subgroup 35 differ from each other only in the rotation angle θ. When the workpieces 2 are randomly piled up, the workpieces 2 may take various postures. Therefore, no priority is set for the rotation angle θ. In other words, the CPU 110 may select the template images 30 one by one in any order from the multiple template images 30 belonging to the image subgroup 35.

[0095] <How to set the upper limit> Fig. 11 is a diagram for explaining an example of a method for setting an upper limit value. As shown in Fig. 11, the CPU 110 operating as the setting unit 14 may set an upper limit value based on a result of a three-dimensional object recognition process using a sample image 20z. The sample image 20z is a distance image generated based on a captured image obtained by the measurement head 400 capturing an image of the randomly piled workpieces 2. One of the input images 20 may be used as the sample image 20z.

[0096] The CPU 110 performs object recognition processing on the sample image 20z in accordance with the flow shown in FIG. 6 without setting an upper limit value (step S50).

[0097] The CPU 110 records the number of candidates and the processing time for each process included in the object recognition process for the sample image 20z (step S51).

[0098] The CPU 110 acquires a target processing time for the object recognition processing (step S52). For example, the CPU 110 may acquire the target processing time from the input device 160. In this way, the user may input a desired target processing time into the input device 160.

[0099] CPU 110 calculates an upper limit value based on the information recorded in step S51 and the target processing time acquired in step S52 (step S53).

[0100] In the flow shown in Fig. 6, the processes that take relatively long processing times are step S12 (pyramid search process), step S14 (detailed search process), and step S18 (detailed alignment process). In step S13 (nearby candidate removal process) between step S12 and step S14, candidates are narrowed down. Similarly, in steps S15 and S17 between step S14 and step S18, candidates are narrowed down. Therefore, CPU 110 calculates the upper limit value as follows.

[0101] CPU 110 calculates the time Ta required to search for one candidate from the number of candidates extracted in step S12 and the processing time of step S12. CPU 110 calculates the time Tb required to process each candidate from the number of candidates processed in step S14 and the processing time of step S14. CPU 110 calculates the time Tc required to process each candidate from the number of candidates processed in step S18 and the processing time of step S18. Each of steps S12, S14, and S18 is an example of the "later processing" of the present disclosure.

[0102] CPU 110 calculates a decay rate Da of the number of candidates in step S13. CPU 110 calculates a decay rate Db of the number of candidates in steps S15 to S17. Step S13 is an example of the "later process" in the present disclosure. In addition, the process including steps S15 to S17 is also an example of the "later process" in the present disclosure.

[0103] The CPU 110 calculates the value of N that satisfies the following formula as the upper limit. Target processing time ≧ N×Ta+N×Da×Tb+N×Da×Db×Tc

[0104] <Modification> 5, each of the multiple template images 30 may be classified into one of multiple image groups having different distances D between the work model 2M and the viewpoint 32. In this case, the image processing unit 10 may preferentially select, from the multiple template images 30, a template image that belongs to an image group in which the distance between the work model 2M and the viewpoint 32 is relatively short.

[0105] 9, loop process L5 is performed for each pixel (x, y) in the first layer image (1) corresponding to the input image 20. In general, when the robot 300 picks up a workpiece 2 located near the side wall of the container 3, there is a possibility that the robot 300 may interfere with the side wall of the container 3. Therefore, the CPU 110 may perform loop process L5 preferentially for positions in the input image 20 that belong to a specific range including the vicinity of the center of the container 3.

[0106] Alternatively, the CPU 110 may change the specific range in which the loop process L5 is executed with priority every time a workpiece 2 is picked up. This allows the robot 300 to pick up the workpieces 2 uniformly from the entire area of ​​the container 3.

[0107] Alternatively, the CPU 110 may set a range including a candidate recognized in the previous object recognition process or the periphery of the candidate as the specific range.

[0108] In the above example, the object recognition process for recognizing the position and orientation of an object in a three-dimensional space has been described. However, the technology disclosed herein may be applied to an object recognition process for recognizing the position and orientation of an object in a two-dimensional space.

[0109] In the above example, the multiple template images 30 are generated using the workpiece model 2M. However, the multiple template images 30 may be generated by photographing the workpiece 2 from various viewpoints. In this case, the workpiece 2 is captured in each template image 30.

[0110] §3 Supplementary Note As described above, the present embodiment includes the following disclosure.

[0111] (Configuration 1) An image processing device (100), an image processing unit (10, 110) that repeatedly performs a search process to search for candidates for an area in which the object (2) exists from an input image (20) by template matching using template images selected in order from a plurality of template images (30) in which an object (2) or a model (2M) corresponding to the object (2) is captured based on image features of the template images (30); The image processing unit (10, 110) terminates the search process in response to the number of the candidates reaching an upper limit.

[0112] (Configuration 2) Each of the plurality of template images (30) is classified into one of a plurality of image groups (34) having different directions of viewpoints (32) with respect to the object (2) or the model (2M); The image processing device (100) according to configuration 1, wherein the image processing unit (10, 110) preferentially selects, from among the plurality of template images (30), a template image (30) that belongs to an image group (35) in which an area of ​​an orthogonal projection of the object (2) or the model (2M) along the direction of the viewpoint (32) is relatively large.

[0113] (Configuration 3) Two or more template images (30) belonging to each of the plurality of image groups (34) are classified into one of a plurality of image subgroups (35) having mutually different distances between the viewpoint (32) and the object (2) or the model (2M); The image processing device (100) according to configuration 2, wherein the image processing unit (10, 110) preferentially selects, from among the two or more template images (30), a template image (30) that belongs to an image subgroup (35) in which the distance is relatively short.

[0114] (Configuration 4) Each of the plurality of template images (30) is classified into one of a plurality of image groups having mutually different distances between the object (2) or the model (2M) and a viewpoint (32); The image processing device (100) according to configuration 1, wherein the image processing unit (10, 110) preferentially selects, from among the plurality of template images (30), a template image (30) that belongs to an image group in which the distance is relatively short.

[0115] (Configuration 5) 5. The image processing device (100) according to any one of configurations 1 to 4, wherein the image processing unit (10, 110) performs the template matching preferentially on a specific range of the input image (20).

[0116] (Configuration 6) 6. The image processing device (100) according to any one of configurations 1 to 5, wherein the upper limit value is set in response to a user input.

[0117] (Configuration 7) A setting unit (14, 110) that sets the upper limit value, 6. The image processing device according to any one of configurations 1 to 5, wherein the setting unit sets the upper limit based on a result of the search process using a sample image (20z) and a target processing time.

[0118] (Configuration 8) The image processing unit (10, 110) further performs at least one subsequent process following the search process, The setting unit (14, 110) calculating a first time required to search for one of the candidates from a result of the search process using the sample image (20z); calculating a second time required for the post-processing for each candidate and a decay rate of the number of candidates due to the post-processing from a result of each of the at least one post-processing using the sample image; The image processing device (100) according to configuration 7, wherein the upper limit value is calculated using the first time, the second time, the attenuation rate, and the target processing time.

[0119] (Configuration 9) 1. An image processing method, comprising: A processor (110) selects one template image (30) from among a plurality of template images (30) in which an object (2) or a model (2M) corresponding to the object (2) is depicted; the processor (110) searches for a candidate region in which the object (2) exists from within the input image (20) by template matching using the one template image (30); The steps of selecting one template image (30) and searching for the candidates are repeated until the number of the candidates reaches an upper limit value, and the image processing method further comprises: The image processing method includes terminating the selection of the one template image and the search for the candidates in response to the number of the candidates reaching the upper limit value.

[0120] (Configuration 10) A program (115) for causing a computer (110) to execute an image processing method, The image processing method includes: selecting one template image (30) from among a plurality of template images (30) depicting an object (2) or a model (2M) corresponding to said object; and searching for a candidate region in which the object (2) exists from within the input image (20) by template matching using the one template image (30). The steps of selecting one template image (30) and searching for the candidates are repeated until the number of the candidates reaches an upper limit value, and the image processing method further comprises: a program (115) that terminates the selection of the one template image (30) and the search for the candidates in response to the number of the candidates reaching the upper limit value.

[0121] Although the embodiment of the present invention has been described, the embodiment disclosed herein should be considered as illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0122] 1 control system, 2 work, 2M work model, 3 container, 10 image processing unit, 11 template image generating unit, 12 head control unit, 13 input image generating unit, 14 setting unit, 15 memory unit, 16 CAD data, 17 captured image, 20 input image, 20z sample image, 30 template image, 31, 31a half line, 32 viewpoint, 34, 34a image group, 35, 35a image subgroup, 100 image processing device, 106 memory card, 110 CPU, 112 main memory, 114 hard disk, 115 program, 116 measuring head interface, 116a image buffer, 118 input interface, 120 display controller, 124 communication interface, 126 data reader / writer, 128 bus, 150 display device, 160 input device, 200 robot controller, 300 robot, 301 End effector, 302 articulated arm, 303 base, 400 measurement head, 401 projection unit, 402 imaging unit.

Claims

1. An image processing device, an image processing unit that repeatedly performs a search process to search for a candidate region in which the object exists from within an input image by template matching using template images selected in order from a plurality of template images in which the object or a model corresponding to the object appears based on image features of the template images, The image processing device, wherein the image processing unit terminates the search process when the number of the candidates reaches an upper limit.

2. Each of the plurality of template images is classified into one of a plurality of image groups having different viewpoint directions with respect to the object or the model; The image processing device according to claim 1 , wherein the image processing unit preferentially selects, from among the plurality of template images, a template image that belongs to an image group in which an area of ​​an orthogonal projection of the object or the model along the direction of the viewpoint is relatively large.

3. The two or more template images belonging to each of the plurality of image groups are classified into any one of a plurality of image subgroups having mutually different distances between the viewpoint and the object or the model; The image processing device according to claim 2 , wherein the image processing unit preferentially selects, from among the two or more template images, a template image that belongs to an image subgroup in which the distance is relatively short.

4. Each of the plurality of template images is classified into one of a plurality of image groups having different distances between the object or the model and a viewpoint; The image processing device according to claim 1 , wherein the image processing unit preferentially selects, from among the plurality of template images, a template image that belongs to an image group in which the distance is relatively short.

5. The image processing device according to claim 1 , wherein the image processing unit performs the template matching with priority on a specific range of the input image.

6. The image processing device according to claim 1 , wherein the upper limit value is set in response to a user input.

7. A setting unit that sets the upper limit value, The image processing device according to claim 1 , wherein the setting unit sets the upper limit based on a result of the search process using a sample image and a target processing time.

8. The image processing unit further performs at least one subsequent process following the search process, The setting unit is calculating a first time required to search for one of the candidates from a result of the search process using the sample image; calculating a second time required for the post-processing for each candidate and a decay rate of the number of candidates due to the post-processing from a result of each of the at least one post-processing using the sample image; The image processing device according to claim 7 , wherein the upper limit value is calculated using the first time, the second time, the attenuation rate, and the target processing time.

9. 1. An image processing method, comprising: A processor selects one template image from among a plurality of template images depicting an object or a model corresponding to the object based on image features of the template image; The processor searches for a candidate region in which the object exists from within the input image by template matching using the one template image, The step of selecting one template image and the step of searching for the candidates are repeated until the number of the candidates reaches an upper limit value, and the image processing method further comprises: the processor terminating the selection of the one template image and the searching for the candidates in response to the number of the candidates reaching the upper limit.

10. A program for causing a computer to execute an image processing method, The image processing method includes: selecting one template image from a plurality of template images depicting an object or a model corresponding to the object based on image features of the template image; searching for a candidate region in which the object exists from within the input image by template matching using the one template image; The step of selecting one template image and the step of searching for the candidates are repeated until the number of the candidates reaches an upper limit value, and the image processing method further includes: terminating the selection of the one template image and the search for the candidates in response to the number of the candidates reaching the upper limit.

Images