Template matching device, template matching method, and template matching system

Abstract

To provide a template-matching device capable of accurate template-matching of an object even in a situation where the posture of the object from the imaging device changes as the imaging device moves.SOLUTION: A template matching device includes: a communication unit that communicates with a storage unit that stores multiple pieces of location information an imaging device capable of picking up images of an object and moving and a template of the object while associating to each other; an acquisition unit that acquires the location information of the imaging device; a prediction unit that performs a series of prediction processing to predict a template used for template matching of the object out of the multiple templates stored in the storage unit based on the location information of the imaging device; and a matching unit that performs template matching by using an input image of the object picked up by the imaging device and the predicted result of the template.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a template matching device, a template matching method, and a template matching system. [Background technology]

[0002] In a production process in a factory, it is sometimes necessary to determine whether a part being picked by an end effector such as a robot hand is the correct part (e.g., a part to be used in the production of an industrial product). In such a determination process, it is necessary to perform the determination process as quickly as possible to avoid reducing the takt time of the production process. A known conventional determination process is, for example, a template matching method, which performs a matching process by comparing a prepared template (e.g., an image) of the part with an image of the part captured by a camera installed in the factory.

[0003] Patent Document 1 discloses a template creation device that creates a set of templates used in an object recognition device that recognizes objects through template matching. The template creation device acquires multiple templates from multiple images of an object in different poses, calculates the similarity of image features between two templates selected from the multiple templates, and performs clustering to divide the multiple templates into multiple groups based on the similarity. The template creation device integrates all templates in each of the multiple groups into a single integrated template, and generates a template set with an integrated template for each group. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-207147 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, the object recognition device performs a hierarchical search by creating a hierarchical template set, performing rough recognition using a low-resolution template set, and then using the results to perform detailed recognition using a high-resolution template set. However, this requires matching processing in at least two stages, i.e., recognition processing using a low-resolution template set and recognition processing using a high-resolution template set, which inevitably increases the processing load of the object recognition device.

[0006] Furthermore, when attempting to apply the technology of Patent Document 1 to the above-described production process in a factory, in which the end effector and camera are moved to capture an image of the part to be picked with the camera in order to determine whether the part to be picked by the end effector is the correct part, the following problem arises. Specifically, if the camera moves as the end effector moves, the way the part appears to the camera (in other words, the posture of the part) changes as the position of the end effector changes. For this reason, unless the position of the end effector (the position of the camera, so to speak) is taken into consideration during template matching, efficient template matching cannot be performed even if a pre-generated template set is used, and the reliability of template matching does not improve.

[0007] The present disclosure has been devised in view of conventional circumstances, and aims to provide a template matching device, a template matching method, and a template matching system that achieve highly accurate template matching of an object even in a situation where the orientation of the object from an imaging device changes as the imaging device moves. [Means for solving the problem]

[0008] The present disclosure provides a template matching device including: a communication unit that communicates with a storage unit that stores position information of an imaging device that is capable of imaging and moving an object and templates of the object in association with each other; an acquisition unit that acquires the position information of the imaging device; a prediction unit that performs a prediction process that predicts a template to be used for template matching of the object from the plurality of templates stored in the storage unit based on the position information of the imaging device; and a matching unit that performs the template matching using an input image of the object imaged by the imaging device and a result of the prediction process of the template.

[0009] The present disclosure also provides a template matching method executed by a template matching device, the template matching method including the steps of: communicating with a storage unit that stores position information of an imaging device that is capable of imaging and moving an object and templates of the object in association with each other; acquiring the position information of the imaging device; performing a prediction process that predicts a template to be used for template matching of the object from the plurality of templates stored in the storage unit, based on the position information of the imaging device; and performing the template matching using an input image of the object imaged by the imaging device and a result of the prediction process of the template.

[0010] The present disclosure also provides a template matching system including: an imaging device capable of imaging and moving an object; and a template matching device connected to and capable of communicating with the imaging device, wherein the template matching device includes: a communication unit that communicates with a storage unit that stores a plurality of positional information of the imaging device and templates of the object in association with each other; an acquisition unit that acquires the positional information of the imaging device; a prediction unit that performs a prediction process that predicts a template to be used for template matching of the object from the plurality of templates stored in the storage unit, based on the positional information of the imaging device; and a matching unit that performs the template matching using an input image of the object imaged by the imaging device and a result of the prediction process of the template. [Effects of the Invention]

[0011] According to the present disclosure, highly accurate template matching of an object can be achieved even under circumstances in which the orientation of the object from the imaging device changes as the imaging device moves. [Brief explanation of the drawings]

[0012] [Figure 1] A simplified diagram showing an example of a picking system configuration [Figure 2] FIG. 1 is a block diagram showing a detailed internal configuration example of a picking system according to a first embodiment. [Figure 3] 1 is a flowchart showing an example of an operation procedure for registering a template by the image processing device according to the first embodiment; [Figure 4] FIG. 10 shows an example of a template registration screen displayed on a display in the first embodiment. [Figure 5] FIG. 10 is a block diagram showing a detailed internal configuration example of a picking system according to a second embodiment. [Figure 6] 10 is a flowchart showing an example of the template matching operation procedure performed by the image processing device according to the second embodiment. [Figure 7] FIG. 10 shows an example of a matching result screen displayed on a display in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Background to the first embodiment) In JP 2016-207147 A, it is assumed that an object recognition device equipped with a template creation device recognizes objects on a conveyor using images captured by a camera, and that the camera itself is installed in a fixed location that cannot be moved relative to a production line, etc. Furthermore, when the object recognition device actually performs matching processing, a template set consisting of integrated templates for each group is used as is.

[0014] However, when attempting to apply the technology of JP 2016-207147 A to the above-described production process in a factory, in which the end effector and camera are moved to capture an image of the part to be picked with the camera in order to determine whether the part to be picked by the end effector is the correct part, the following problem arises. Specifically, if the camera moves as the end effector moves, the way the part appears to the camera (in other words, the posture of the part) changes as the position of the end effector changes. For this reason, unless the position of the end effector (in other words, the position of the camera) is taken into consideration during template matching, efficient template matching cannot be performed even if a pre-generated template set is used, and the reliability of template matching does not improve.

[0015] Therefore, in the following first embodiment, examples of a template registration device, a template registration method, and a template registration system will be described that register high-precision templates of an object that can be used for template matching even in a situation where the orientation of the object from the image capturing device changes as the image capturing device moves.

[0016] Hereinafter, with appropriate reference to the accompanying drawings, embodiments specifically disclosing a template registration device, template registration method, and template registration system according to the present disclosure will be described in detail. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

[0017] (First embodiment: overview) In the first embodiment, a use case will be described in which a template required for template matching is registered when determining, by template matching, whether a part to be picked by an end effector such as a robot hand (e.g., a part used in the production of an industrial product) can be correctly recognized in a production process in a factory, for example. A template registration device (e.g., an image processing device) according to the present disclosure acquires information (see below) based on an input image of an object captured by an imaging device that is capable of capturing and moving the object, and position information of the imaging device, and associates the information based on the input image of the object (see below) with the position information of the imaging device as a template to be used in template matching, and registers the information based on the input image of the object (see below) in a storage unit.

[0018] (Embodiment 1: Details) FIG. 1 is a diagram simply illustrating an example configuration of a picking system 100. FIG. 2 is a block diagram illustrating an example detailed internal configuration of the picking system 100 according to the first embodiment. As illustrated in FIG. 2, the picking system 100 (an example of a template registration system) includes an actuator ACT1, a camera CAM1, an image processing device 10, an operation device 20, a display 30, and a template registration device 40. The actuator ACT1 and the image processing device 10, the camera CAM1 and the image processing device 10, the image processing device 10 and the operation device 20, the image processing device 10 and the display 30, and the image processing device 10 and the template registration device 40 are connected to each other so as to enable input and output (transmission and reception) of data signals.

[0019] The positional relationship between the camera CAM1 and the object OB1 based on the control of the actuator ACT1 will be described with reference to Fig. 1. The description of Fig. 1 is applicable not only to the first embodiment but also to the second embodiment described later.

[0020] In the following description, an object OB1 is an object to be picked by the end effector EF1 of the picking system 100 installed in a factory, such as an industrial part or an industrial product. If it is an industrial part, it is moved to another lane (production line) after being picked to assemble a finished product. If it is an industrial product, it is stored in a box such as a cardboard box after being picked. It goes without saying that the type of object OB1 is not limited to the above-mentioned industrial part or industrial product.

[0021] As shown in FIG. 1, an actuator ACT1 controls the positional relationship between camera CAM1 and an object OB1, which is fixedly placed on a predetermined location in a factory (e.g., a surface plate FL1), by controlling the camera CAM1 so that it can move three-dimensionally. FIG. 1 illustrates a Cartesian coordinate system consisting of X, Y, and Z axes to define three-dimensional positions (coordinates). Note that the origin of this Cartesian coordinate system is not shown in FIG. 1. The X and Y axes form a horizontal plane in real space, and the Z axis indicates a direction parallel to the direction of gravity, which is perpendicular to the horizontal plane. The X axis is the left-right direction on the paper in FIG. 1, and the Y axis is the direction perpendicular to the X axis (in other words, the depth direction of the Cartesian coordinate system in FIG. 1).

[0022] The type of surface plate FL1 is not particularly limited as long as it represents a horizontal surface such as a floor surface. An object OB1, which is the subject of camera CAM1, is fixedly placed on this surface plate FL1 and picked up by an end effector EF1 (e.g., a robot hand) that moves together with camera CAM1.

[0023] The actuator ACT1 controls the movement of each of the following arms: an X arm XDR1 movable along an X rail XRL1 extending in the X-axis direction, a Y arm YDR1 movable along a Y rail (not shown) extending in the Y-axis direction, and a Z arm ZDR1 movable along a Z rail ZRL1 extending in the Z-axis direction. In other words, the actuator ACT1 can recognize, maintain, and change the three-dimensional positions (coordinates) of the end effector EF1 and the camera CAM1 based on control commands sent from the image processing device 10, and constantly or periodically sends coordinates (an example of position information) indicating the three-dimensional positions of the end effector EF1 and the camera CAM1 to the image processing device 10.

[0024] The end effector EF1 is a robot hand provided at the tip of a robot arm (not shown) deployed, for example, in correspondence with the picking system 100, and can move closer to the object OB1 under the control of the actuator ACT1 and pick up the object OB1.

[0025] The camera CAM1 (an example of an imaging device) is arranged in the immediate vicinity of the end effector EF1, and moves in pair with the end effector EF1 to approach the object OB1 under the control of the actuator ACT1, and can capture an image of the object OB1 included in the angle of view (in other words, the field of view range FV1). The camera CAM1 captures the object OB1, which is the subject within the field of view range FV1, at a predetermined frame rate, and sends the captured image (an example of an input image) of the object OB1 obtained each time it is captured to the image processing device 10.

[0026] The image processing device 10 (an example of a template registration device) is configured by a computer capable of executing predetermined processing (see FIG. 3 ) using position information from the actuator ACT1 and captured images (input images) of the object OB1 from the camera CAM1. For example, the image processing device 10 may be a PC (Personal Computer) or a dedicated hardware device specialized for the predetermined processing. By performing the predetermined processing, the image processing device 10 registers (saves) some of the captured images input from the camera CAM1 in the template registration device 40 as templates to be used for template matching of the object OB1 (see embodiment 2). Furthermore, the image processing device 10 generates a template registration screen WD1 (see FIG. 4 ) for template registration and displays it on the display 30, so that the decision on whether to register the template is up to the user (e.g., the person performing the operation related to template registration). Therefore, the image processing device 10 may decide to register the template to be registered (see FIG. 4 ) or discard the template, based on a user operation on the template registration screen WD1. Furthermore, the image processing device 10 may generate a control command for commanding the movement of the end effector EF1 and the camera CAM1 according to the template matching result and a predetermined movement path map (see FIG. 4), and send the control command to the actuator ACT1.

[0027] The operation device 20 is an interface that detects input of a user operation and is configured by, for example, a mouse, a keyboard, or a touch panel. When the operation device 20 receives a user operation, it generates a signal based on the operation and sends it to the image processing device 10.

[0028] The display 30 is a device that outputs (displays) the display screen (see FIG. 4) generated by the image processing device 10, and is configured by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) device.

[0029] The template registration device 40 (an example of a storage unit) is, for example, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The template registration device 40 non-temporarily stores data of a template (image) determined to be registered by the image processing device 10 in association with position information (three-dimensional position) of the camera CAM1 that captured the image corresponding to the template. The data of each template stored in the template registration device 40 includes at least the template (image) data or compressed data of the template, position information of the camera CAM1 that captured the image corresponding to the template, and position information and feature amounts of multiple feature points extracted from the template by a feature extraction unit 121 (see below). The template (image) data or compressed data of the template (e.g., thumbnail) and the feature amounts of multiple feature points extracted from the template by the feature extraction unit 121 are examples of information based on the input image. The feature points and feature amounts of the template may be registered or may be recalculated and acquired each time they are referenced.

[0030] Here, the internal configuration of the image processing device 10 will be described in detail.

[0031] The image processing device 10 includes at least a communication interface 11, a processor 12, and a memory 13. The communication interface 11, the processor 12, and the memory 13 are connected via a data transmission bus (not shown) so as to enable input and output of data signals among them.

[0032] The communication interface 11 (an example of an acquisition unit or communication unit) is a communication circuit that inputs and outputs (transmits and receives) data signals between the actuator ACT1 and the image processing device 10, between the camera CAM1 and the image processing device 10, between the image processing device 10 and the operation device 20, between the image processing device 10 and the display 30, and between the image processing device 10 and the template registration device 40. In the accompanying drawings, the term "interface" is abbreviated as "I / F." The communication interface 11 constantly or periodically receives position information from the actuator ACT1 and temporarily stores it in the processor 12 or the memory 13. The communication interface 11 receives captured images (e.g., images of the object OB1) input from the camera CAM1 and temporarily stores them in the memory 13. The communication interface 11 outputs a template registration screen (see FIG. 4 ) generated by the processor 12 to the display 30. The communication interface 11 sends control commands generated by the processor 12 regarding the movement of the end effector EF1 and the camera CAM1 to the actuator ACT1.

[0033] The processor 12 is, for example, a central processing unit (CPU), a graphical processing unit (GPU), a digital signal processor (DSP), or a field programmable gate array (FPGA). The processor 12 functions as a controller that manages the overall operation of the image processing device 10, and performs control processing to oversee the operation of each unit of the image processing device 10, data input / output processing between each unit of the image processing device 10, data calculation processing, and data storage processing. The processor 12 operates according to programs and control data stored in the memory 13, uses the memory 13 during operation, and temporarily stores data or information generated or acquired by the processor 12 in the memory 13 or sends it to an external device (e.g., the display 30, the template registration device 40, or the actuator ACT1) via the communication interface 11. For example, the processor 12 can functionally realize a feature extraction unit 121, a feature matching unit 122, an MoS calculation unit 123, an MaS calculation unit 124, a template registration determination unit 125, and a control unit 126 in accordance with the program and control data stored in the memory 13.

[0034] The feature extraction unit 121 targets the captured images input from the camera CAM1 and extracts, for each captured image, a plurality of feature points related to the object OB1 (i.e., feature-like positions on the image where, for example, rotational change, scale change, or brightness change is greatest in the captured image showing the object OB1). This extraction method can utilize a known algorithm such as SIFT (Scale-Invariant Feature Transform). The feature extraction unit 121 extracts a plurality of feature points from one input captured image and temporarily stores the extraction results (e.g., point cloud data indicating the respective positions of the plurality of feature points in the captured image and feature amounts for each feature point; the same applies below) in the processor 12 or the memory 13. Hereinafter, the extracted results of the feature points (i.e., position information and feature amounts of the feature points) may be collectively referred to as "features."

[0035] The feature matching unit 122 acquires a template corresponding to the position information from the actuator ACT1 from the template registration device 40 via the communication interface 11. The feature matching unit 122 performs template matching of the object OB1 using the extraction results of feature points on the captured image by the feature extraction unit 121 and the template obtained from the template registration device 40. For example, the feature matching unit 122 separates the extraction results of feature points on the captured image into feature points and feature amounts on the template from the template registration device 40 and feature points and feature amounts newly extracted on the captured image in a distinguishable manner.

[0036] The MoS calculation unit 123 (an example of an index calculation unit) calculates an MoS (Moving Score) as an example of a first index indicating the difference between the position of the camera CAM1 when the template was captured and the position of the camera CAM1 when the captured image input from the camera CAM1 was captured (in other words, the degree of positional similarity). This MoS is used to determine whether the template registration determination unit 125 should register, as a template, an image (input image) captured by the camera CAM1 when the camera CAM1 is located at the intended registration position (see FIG. 4). In other words, even if an image highly similar to an already registered template is input, the image processing device 10 calculates the MoS to avoid registering such an image as a template as much as possible. The MoS calculation unit 123 calculates the MoS using, for example, equation (1).

[0037]

number

[0038] In equation (1), pt f is the three-dimensional coordinate (x f ,y f ,z f) where f indicates the number of the position (planned registration position) where the fth frame (i.e., the captured image) is to be registered as a template, and corresponds to, for example, position P6 shown in FIG. tm is the three-dimensional coordinate (x tm ,y tm ,z tm ) where tm indicates the number when the most recent template was registered (for example, a number indicating the order of registration), and corresponds to position P5 shown in FIG. 4, for example. That is, MoS is calculated as the reciprocal of the difference between the intended registration position and the immediately preceding template registration position (see FIG. 4). Therefore, when the difference between the immediately preceding template registration position of the end effector EF1 and camera CMA1 and the current intended registration position is large, MoS is small, and the template registration determination unit 125 determines (decides) to register the captured image at the current intended registration position as a template. On the other hand, when the difference between the immediately preceding template registration position of the end effector EF1 and camera CMA1 and the current intended registration position is small, MoS is large, and the template registration determination unit 125 determines (decides) not to register the captured image at the current intended registration position as a template.

[0039] The MaS calculation unit 124 (an example of an index calculation unit) calculates MaS (Matching Score) as an example of a second index indicating the image correlation (in other words, the degree of similarity of the images) between the features of the object OB1 in the template and the features of the object OB1 in the captured image. This MaS is used by the template registration determination unit 125 to determine whether or not to register the captured image (input image) captured by the camera CAM1 when the camera CAM1 is present at the intended registration position (see FIG. 4). The MaS calculation unit 124 calculates MaS using, for example, the calculation formula (2). M in formula (2) j follows the condition of equation (3). In equation (2), FS j denotes the matching score of the jth feature.

[0040]

number

[0041]

number

[0042] In other words, MaS indicates the matching ratio between the features on the template registered corresponding to the immediately previous template registration position (see FIG. 4) and the features on the captured image input from the camera CAM1 corresponding to the current planned registration position (see FIG. 4). Therefore, when there is a large difference between the features on the template at the immediately previous template registration position of the end effector EF1 and the camera CMA1 and the features on the input captured image at the current planned registration position, the number of matching features is small, and MaS tends to be small (in other words, M j On the other hand, when the difference between the features of the template at the immediately preceding template registration position of the end effector EF1 and the camera CMA1 and the features of the template at the current registration position is small, the number of matching features is large, so MaS tends to be large (in other words, M j does not become 0), the template registration determination unit 125 determines (decides) that the captured image at the current intended registration position should not be registered as a template.

[0043] The template registration determination unit 125 (an example of a registration determination unit) determines (decides) whether or not to register a captured image of the current intended registration position as a template, based on the result of comparing the MoS calculation result by the MoS calculation unit 123 with a threshold value Th1 stored in the memory 13, or the result of comparing the MaS calculation result by the MaS calculation unit 124 with a threshold value Th2 stored in the memory 13. When the template registration determination unit 125 (an example of a control unit) determines (decides) that the captured image of the current intended registration position is to be registered as a template, the template registration determination unit 125 at least associates data of the captured image with position information (intended registration position) of the camera CAM1 that captured the captured image, and extraction results including positions and feature amounts of multiple feature points extracted from the captured image, and registers (saves) these in the template registration device 40. Note that the template registration determination unit 125 may further associate a distance difference value between the previous template registration position and the current intended registration position with the movement direction of the camera CAM1, and registers (saves) these in the template registration device 40.

[0044] The control unit 126 is responsible for the integrated control management of various processes executed by the processor 12. The control unit 126 generates control commands related to the movement of the end effector EF1 and camera CAM1 by the movement amounts in accordance with a movement path map MP1 (see FIG. 4) stored in advance in the memory 13, and sends these to the actuator ACT1 via the communication interface 11. The actuator ACT1 controls the end effector EF1 and camera CAM1 to move three-dimensionally based on the control commands sent from the image processing device 10. The control unit 126 generates a template registration screen WD1 related to template registration, and displays it on the display 30 via the communication interface 11.

[0045] The memory 13 includes at least a RAM (Random Access Memory) and a ROM (Read Only Memory), for example, and temporarily stores programs and control data required to execute operations of the image processing device 10, as well as data generated or acquired by each unit of the image processing device 10 while it is executing a process. The RAM is, for example, a work memory used by each unit of the image processing device 10 while it is executing a process. The ROM stores, in advance, programs and control data that define the processing of each unit of the image processing device 10. The memory 13 may further include, for example, a flash memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive), and may store the same data as the template registered (saved) in the template registration device 40 (see FIG. 4).

[0046] Next, an example of an operational procedure for registering a template by the image processing device 10 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of an operational procedure for registering a template by the image processing device 10 according to the first embodiment. The processing shown in Fig. 3 is mainly executed by the processor 12 of the image processing device 10.

[0047] In FIG. 3, the processor 12 initializes a variable i (step St1). i is a variable indicating the number of a registered template, and takes on an integer value equal to or greater than 0 up to tm. In other words, tm indicates the number of the template registered immediately before. The processor 12 inputs and acquires a captured image (input image) from the camera CAM1 (step St2), and further inputs and acquires at least the position information of the camera CAM1 from the actuator ACT1 (step St3). The processor 12 extracts feature points from the captured image (input image) acquired in step St2 (step St4).

[0048] If i is not greater than zero (0) (step St5, NO), that is, if i=0 (no template has been registered yet), the processor 12 increments i (step St10) and registers the captured image (input image) acquired in step St2 as a template in the template registration device 40 (step St11).

[0049] On the other hand, if i is greater than zero (0) (step St5, YES), the processor 12 performs template matching of the object OB1 between the captured image (input image) of the camera CAM1 acquired in step St2 and the template, which is the captured image immediately before (i.e., the tmth) (step St6). The processor 12 calculates MaS as an example of an index indicating the image correlation (in other words, the image similarity) between the features of the object OB1 in the template and the features of the object OB1 in the captured image (step St7). The processor 12 also calculates MoS as an example of a first index indicating the difference (in other words, the position similarity) between the position of the camera CAM1 when the template was captured and the position of the camera CAM1 when the captured image input from the camera CAM1 was captured (step St8). The processing of steps St7 and St8 may be performed in any order.

[0050] The processor 12 determines whether or not at least one of the conditions that the MaS calculated in step St7 is smaller than the threshold value Th2 and the MoS calculated in step St8 is smaller than the threshold value Th1 is satisfied (step St9). If it is determined that the MaS is larger than the threshold value Th2 and the MoS is larger than the threshold value Th1 (step St9, NO), the process of the processor 12 proceeds to step St12.

[0051] On the other hand, if the processor 12 determines that MaS is smaller than the threshold value Th2 or that MoS is smaller than the threshold value Th1 (step St9, YES), it increments i (step St10) and registers the image (input image) captured by the camera CAM1 in step St2 as the i-th template (i.e., the i before the increment) in the template registration device 40 (step St11). If the processor 12 determines that the actuator ACT1 is causing the end effector EF1 to continue operating (e.g., moving) (step St12, YES), the processing by the processor 12 returns to step St2. On the other hand, if the processor 12 determines that the actuator ACT1 is not causing the end effector EF1 to continue operating (e.g., moving) (step St12, NO), the processor 12 ends the template registration processing when the patrol along the movement path map MP1 (see FIG. 4) is completed. This ends the processing by the processor 12 shown in FIG. 3.

[0052] At the time of step St11, the processor 12 may display the template registration screen WD1 shown in FIG. 4 on the display 30, thereby leaving it to the user to decide whether or not to register a template for the current planned registration point.

[0053] 4 is a diagram showing an example of a template registration screen WD1 displayed on the display 30 in the first embodiment. The template registration screen WD1 is generated, for example, by the control unit 126 of the processor 12. The template registration screen WD1 has a travel route map display area SD1, a display area SD2 for templates to be registered, a display area SD3 for registered templates, a register button BT1, and a discard button BT2.

[0054] The movement path map display area SD1 displays a movement path map MP1 that shows the movement path (movement history) of the pair of end effector EF1 and camera CAM1. Positions p1, p2, p3, p4, and p5 each indicate a position where template registration was performed when the pair of end effector EF1 and camera CAM1 was previously located. Position p6 indicates a planned registration position (i.e., a position that is the subject of a determination as to whether or not to register an image captured by camera CAM1 at position p6 as a template, in relation to the template registered at the immediately preceding registration position (e.g., position p5)). Position p7 indicates the current (latest) position of the pair of end effector EF1 and camera CAM1. Coordinates indicating the three-dimensional position of each of positions p1 to p7 are input from actuator ACT1 to image processing device 10.

[0055] The to-be-registered template display area SD2 displays a captured image IMG6 to be registered as a template showing the captured image (input image) captured by camera CAM1 at the to-be-registered position (see above) together with a product number (e.g., "XX38YZ1X"). This to-be-registered captured image IMG6 distinguishably displays, among all feature points extracted by the feature extraction unit 121 of the processor 12, newly extracted feature points (see circles) obtained by the feature matching unit 122 and feature points extracted in the immediately preceding template (see triangles). Therefore, by viewing this to-be-registered template display area SD2, the user can visually determine the number or proportion of newly extracted feature points in the to-be-registered captured image IMG6, and can easily decide whether or not to register the to-be-registered captured image IMG6 as a template.

[0056] The registered template display area SD3 displays template registration images TPL1, TPL2, ..., when templates were registered at past registration positions (positions p1 to p5 in the example of FIG. 4). Each of the template registration images TPL1, TPL2, ... displays the corresponding template IMG1, IMG2, ..., in association with the position of the camera CAM1 that captured the template (i.e., the three-dimensional coordinates of the corresponding registration position). The example of FIG. 4 shows template registration image TPL1 including template IMG1 in which an image captured by camera CAM1 at three-dimensional coordinates P1Z (0,0,0) corresponding to position p1 is registered, and template registration image TPL2, ..., including template IMG2 in which an image captured by camera CAM1 at three-dimensional coordinates P2Z (1,0,0) corresponding to position p2 is registered. Although not shown in FIG. 4, the user can display other registered template registration images by appropriately dragging scroll bars SCB1 and SCB2 to scroll, etc.

[0057] The registration button BT1 is a button that is pressed by a user operation when registering the captured image IMG6 to be registered that is displayed in the template display area SD2 to be registered as a template. That is, the processor 12 of the image processing device 10 can register the captured image IMG6 to be registered as a template by detecting that the registration button BT1 has been pressed by a user operation.

[0058] The discard button BT2 is a button that is pressed by a user operation when the user does not want to register the captured image IMG6 to be registered that is displayed in the template display area SD2 to be registered as a template. In other words, by detecting that the discard button BT2 has been pressed by a user operation, the processor 12 of the image processing device 10 can avoid registering the captured image IMG6 to be registered as a template.

[0059] As described above, in the picking system 100 according to the first embodiment, the image processing device 10 includes a communication interface 11 that acquires an input image of the object OB1 captured by the camera CAM1, which is capable of capturing an image of the object OB1, and position information of the camera CAM1, and a control unit 126 that associates the position information of the camera CAM1 with the information based on the input image of the object OB1 as a template to be used for template matching and registers the information based on the input image of the object OB1 in the template registration device 40. The information based on the input image is, for example, template (image) data or compressed data of the template (e.g., a thumbnail) and feature amounts of a plurality of feature points extracted from the template by the feature extraction unit 121. This allows the image processing device 10 to register a highly accurate template of the object that can be used for template matching, even in a situation where the orientation of the object OB1 as viewed from the camera CAM1 changes as the camera CAM1 moves.

[0060] Furthermore, the control unit 126 associates the position information on the movement path map MP1 along which the camera CAM1 moves with information based on the input image of the object OB1, and registers the associated information in the template registration device 40. This enables the image processing device 10 to register, as a template, the position information of the camera CAM1, which may change as the camera CAM1 moves on the predetermined movement path map MP1, and the information based on the input image of the object OB1 in association with each other.

[0061] The image processing device 10 also includes a template registration determination unit 125 that determines whether a template can be registered. The template registration determination unit 125 determines whether a template can be registered for each piece of position information of the camera CAM1. This allows the image processing device 10 to register many types of templates with different postures, taking into account that the appearance of the object OB1 (in other words, its posture) changes each time the camera CAM1 moves, and contributes to improving the accuracy of template matching.

[0062] The image processing device 10 also includes a feature extraction unit 121 that extracts features of the object OB1 from the input image, a feature matching unit 122 that performs a matching process between the features of the object OB1 that appear in the template stored in the template registration device 40 and the extracted feature points of the object OB1 that appear in the input image, and a template registration determination unit 125 that determines whether or not the template can be registered based on the matching result of the feature matching unit 122 and position information of the camera CAM1. As a result, the image processing device 10 can determine with high accuracy whether or not the template can be registered by comparing the features of the object OB1 that appear in the input image and the template.

[0063] The image processing device 10 also includes an index calculation unit that calculates a first index (e.g., MoS) indicating the difference between the position of the camera CAM1 when the template was captured and the position of the camera CAM1 when the input image was captured, and a second index (e.g., MaS) indicating the image correlation between the features of the object OB1 in the template and the features of the object OB1 in the input image. Here, the index calculation unit corresponds to, for example, the MoS calculation unit 123 and the MaS calculation unit 124. The template registration determination unit 125 determines whether or not to register the template based on the relationship between at least one of the calculation results of the first index and the second index and a threshold value (e.g., threshold value Th1 or threshold value Th2) stored in advance in the memory 13. This allows the image processing device 10 to appropriately determine whether or not to register the input image as a template using at least one of the MoS and MaS.

[0064] Furthermore, when it is determined that at least one of the calculation results of the first index and the second index (for example, at least one of MoS and MaS) is less than a threshold (for example, threshold Th1 or threshold Th2), the template registration determination unit 125 determines to register the input image as a template in the template registration device 40. As a result, when the index is, for example, MoS, if MoS is less than threshold Th1, the difference between the previous template registration position of camera CMA1 and the current planned registration position is large, so by newly registering the input image as a template, the image processing device 10 can avoid accuracy degradation caused by using a previously registered template for template matching. Furthermore, when the index is, for example, MaS, if MaS is less than threshold Th2, the difference between the previous template registration position of camera CMA1 and the current planned registration position is large, so by newly registering the input image as a template, the image processing device 10 can avoid accuracy degradation caused by using a previously registered template for template matching.

[0065] Furthermore, when it is determined that at least one of the calculation results of the first index and the second index (for example, at least one of MoS and MaS) is equal to or greater than a threshold value (for example, threshold value Th1 or threshold value Th2), the template registration determination unit 125 determines not to register the input image as a template in the template registration device 40. As a result, when the index is, for example, MoS, if MoS is equal to or greater than threshold value Th2, the difference between the previous template registration position of camera CMA1 and the current planned registration position is small, and therefore the image processing device 10 can reuse a previously registered template to perform template matching with high accuracy. When the index is, for example, MaS, if MaS is equal to or greater than threshold value Th2, the difference between the previous template registration position of camera CMA1 and the current planned registration position is small, and therefore the image processing device 10 can reuse a previously registered template to perform template matching with high accuracy.

[0066] Furthermore, the template is an input image of the object OB1 captured by the camera CAM1 at the immediately previous registration position. This allows the image processing device 10 to use, as a template, the input image of the object OB1 captured at the immediately previous registration position where the movement amount of the camera CAM1 is not considered to be large, and therefore, it is possible to appropriately determine whether or not the input image captured at the current position of the camera CAM1 may be registered as a template.

[0067] The image processing device 10 also includes a feature extraction unit 121 that extracts features of the object OB1 from the input image. The template registration determination unit 125 displays the input image on the display 30, in which the features of the object OB1 extracted in the template and the features of the object OB1 newly extracted in the input image are colored in different colors. This allows the user to visually confirm the newly extracted feature points by comparing the feature points extracted in the input image at the intended registration position with the feature points in the template, and thus allows the user to appropriately determine whether or not to register the input image at the intended registration position as a template.

[0068] Furthermore, the template registration determination unit 125 displays a movement path map MP1 of camera CAM1, which indicates the registration position of at least one template, on the display 30. This allows the user to visually confirm the movement path along the movement path map MP1 of camera CAM1 in order to register the template, and the position information of the point where the template was registered.

[0069] Furthermore, for each template registered in the template registration device 40, the template registration determination unit 125 associates the template with position information of the camera CAM1, which indicates the registration position of the template, and displays the associated template on the display 30. This allows the user to visually check not only a list of previously registered templates, but also the posture of the object OB1 reflected in each template.

[0070] (Background to the second embodiment) In JP 2016-207147 A, the object recognition device performs a hierarchical search by creating a hierarchical template set, performing rough recognition using a low-resolution template set, and then using the results to perform detailed recognition using a high-resolution template set. However, matching processing must be performed in at least two stages, such as recognition processing using a low-resolution template set and recognition processing using a high-resolution template set, which inevitably increases the processing load of the object recognition device.

[0071] Furthermore, when attempting to apply the technology of JP 2016-207147 A to the above-described production process in a factory, in which the end effector and camera are moved to capture an image of the part to be picked with the camera in order to determine whether the part to be picked by the end effector is the correct part, the following problem arises. Specifically, if the camera moves as the end effector moves, the way the part appears to the camera (in other words, the posture of the part) changes as the position of the end effector changes. For this reason, unless the position of the end effector (in other words, the position of the camera) is taken into consideration during template matching, efficient template matching cannot be performed even if a pre-generated template set is used, and the reliability of template matching does not improve.

[0072] Therefore, in the following second embodiment, examples of a template matching device, a template matching method, and a template matching system that achieve highly accurate template matching of an object even in a situation where the orientation of the object from the imaging device changes as the imaging device moves will be described.

[0073] (Embodiment 2: Overview) In the second embodiment, an example will be described in which, for example, in a production process in a factory, a template registered by the method described in the first embodiment is used to determine by template matching whether a part (an example of an object OB1) to be picked by an end effector EF1 such as a robot hand is the correct part (for example, a part to be used in the production of an industrial product). The template matching device according to the present disclosure communicates with a storage unit that stores position information of an imaging device that can image and move an object and a plurality of templates of the object in association with each other, acquires the position information of the imaging device, and performs a prediction process that predicts a template to be used for template matching of the object from the plurality of templates stored in the storage unit based on the position information of the imaging device, and performs template matching using an input image of the object imaged by the imaging device and the prediction result of the template.

[0074] (Embodiment 2: Details) Fig. 5 is a block diagram showing a detailed example of the internal configuration of a picking system 100A according to embodiment 2. As shown in Fig. 5, the picking system 100A (an example of a template matching system) includes an actuator ACT1, a camera CAM1, an image processing device 10A, an operation device 20, a display 30, and a template registration device 40. As in embodiment 1, the actuator ACT1 and the image processing device 10A, the camera CAM1 and the image processing device 10A, the image processing device 10A and the operation device 20, the image processing device 10A and the display 30, and the image processing device 10A and the template registration device 40 are connected to each other so as to enable input and output (transmission and reception) of data signals.

[0075] In the description of the second embodiment, the same components as those of the picking system 100 according to the first embodiment are given the same reference numerals, and the description thereof will be simplified or omitted, and only the differences will be described.

[0076] The image processing device 10A (an example of a template matching device) is configured by a computer capable of executing predetermined processing (see FIG. 6) using position information from the actuator ACT1 and a captured image (input image) of the object OB1 from the camera CAM1. For example, the image processing device 10 may be a PC (Personal Computer) or a dedicated hardware device specialized for the predetermined processing. By performing the predetermined processing, the image processing device 10A predicts a template to be used for template matching of the captured image input from the camera CAM1 and performs template matching using the predicted template and the input captured image. The image processing device 10 also generates a matching result screen WD2 (see FIG. 7) showing the results of the template matching process and displays it on the display 30. The image processing device 10A may also generate control commands for instructing the movement of the end effector EF1 and the camera CAM1 according to the template matching results and a predetermined movement path map (see FIG. 7), and send the control commands to the actuator ACT1.

[0077] The template registration device 40 (an example of a storage unit) is, for example, a flash memory, HDD, or SSD. The template registration device 40 non-temporarily stores data of templates (images) registered to be used in template matching by the method described in the first embodiment, in association with positional information (three-dimensional position) of the camera CAM1 that captured the image corresponding to the template. The data of each template stored in the template registration device 40 includes at least the template (image) data or compressed data of the template, the positional information and feature amounts of the camera CAM1 that captured the image corresponding to the template, and positional information of a plurality of feature points extracted from the template by the feature extraction unit 121.

[0078] Here, the internal configuration of the image processing device 10A will be described in detail.

[0079] The image processing device 10A includes at least a communication interface 11, a processor 12A, and a memory 13. The communication interface 11, the processor 12A, and the memory 13 are connected via a data transmission bus (not shown) so as to enable input and output of data signals among them.

[0080] The processor 12A is, for example, a CPU, a GPU, a DSP, or an FPGA. The processor 12A functions as a controller that manages the overall operation of the image processing device 10A, and performs control processing to oversee the operation of each unit of the image processing device 10A, data input / output processing between each unit of the image processing device 10A, data calculation processing, and data storage processing. The processor 12A operates according to programs and control data stored in the memory 13, and uses the memory 13 during operation to temporarily store data or information generated or acquired by the processor 12A in the memory 13 or send it to an external device (e.g., the display 30, the template registration device 40, or the actuator ACT1) via the communication interface 11. For example, the processor 12A can functionally implement a feature extraction unit 121, a feature matching unit 122, a control unit 126, a position fitting unit 127, a template prediction unit 128, and a template update determination unit 129 according to the programs and control data stored in the memory 13.

[0081] The position fitting unit 127 performs a fitting process to approximate the position of the object OB1 reflected in the captured image (input image) captured at the current position (see FIG. 7) of the camera CAM1 based on the matching result (see FIG. 7) by the feature matching unit 122. The position of the object OB1 is, for example, a circular range having a predetermined diameter centered at the center position of the object OB1. The result of this fitting process is referred to, for example, when the control unit 126 generates a matching result screen WD2.

[0082] The template prediction unit 128 (an example of a prediction unit) predicts a template for template matching to be used by the feature matching unit 122 from among multiple templates registered in the template registration device 40, based on either or both of position information consisting of the three-dimensional coordinates of the camera CAM1 (i.e., the amount of movement of the camera CAM1) constantly or periodically sent from the actuator ACT1 and the result of the previous (immediately preceding) matching process by the feature matching unit 122. For example, based on position information consisting of the three-dimensional coordinates of the camera CAM1, the template prediction unit 128 predicts a registered template corresponding to the position information as a template for template matching. Furthermore, the template prediction unit 128 calculates MaS (see embodiment 1) using the result of the previous (immediately preceding) matching process, and if it determines that this MaS is equal to or greater than a threshold Th3 stored in the memory 13, for example, predicts the template used by the feature matching unit 122 for template matching as the template for template matching. Note that the template prediction method used by the template prediction unit 128 is not limited to the above-described method.

[0083] The template update determination unit 129 (an example of an update determination unit) determines whether to use a template specified by the operation device 20 based on a user operation or a template predicted by the template prediction unit 128 as a template to be used in the next matching process by the feature matching unit 122. For example, in the case of a frame from the first camera CAM1, the template update determination unit 129 determines to use a template specified by the operation device 20 based on a user operation as a template to be used in the next matching process by the feature matching unit 122. In the case of a frame from the second or subsequent camera CAM1, the template update determination unit 129 determines whether to use a template predicted by the template prediction unit 128 as a template to be used in the next matching process by the feature matching unit 122.

[0084] Next, an example of the operation procedure of template matching by the image processing device 10A according to Embodiment 2 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the operation procedure of template matching by the image processing device 10A according to Embodiment 2. The processing shown in Fig. 6 is mainly executed by the processor 12A of the image processing device 10A.

[0085] 6, the processor 12A inputs and acquires a captured image (input image) from the camera CAM1 (step St21), and further inputs and acquires at least the position information of the camera CAM1 from the actuator ACT1 (step St22). The processor 12A extracts features of the captured image (input image) acquired in step St21 (step St23).

[0086] If the frame is from the first camera CAM1 (step St24, YES), the processor 12A acquires the template selected by the user operation from among the registered templates at the registered position on the travel path map MP2 displayed on the matching result screen WD2 (step St25). After step St25, the processing of the processor 12A proceeds to step St27.

[0087] On the other hand, if the frame is not from the first camera CAM1 (that is, if the frame is from the second or subsequent camera CAM1) (step St24, NO), the processor 12A predicts a template for template matching to be used by the feature matching unit 122 from among the multiple templates registered in the template registration device 40, based on the three-dimensional coordinates of the camera CAM1 (that is, the amount of movement, which is the amount of change in position information) and the result of the previous (immediately preceding) matching process by the feature matching unit 122 (step St26). The processor 12A reads out and acquires the predicted template (that is, the registered template) from the template registration device 40 (step St26).

[0088] The processor 12A performs template matching of the object OB1 using the captured image (input image) acquired in step St23 and the template prediction result (i.e., the registered template) acquired in step St26 (step St27). The processor 12A performs fitting processing to approximate the position of the object OB1 based on the template matching processing result (matching result) of step St27 (step St28). The processor 12A displays the template matching processing result and the fitting processing result on a matching result screen WD2.

[0089] The processor 12A adjusts and determines the movement amounts of the end effector EF1 and the camera CAM1 based on the result of the fitting process in step St28 (step St29). Furthermore, the processor 12A generates a control command for controlling the movement of the end effector EF1 and the camera CAM1 and sends it to the actuator ACT1 via the communication interface 11 (step St29). Based on the control command from the image processing device 10, the actuator ACT1 controls the movement of the end effector EF1 and the camera CAM1 by the movement amount determined by the control command.

[0090] If the processor 12A determines that the movement operation of the end effector EF1 and the camera CAM1 by the actuator ACT1 is continuing (step St30, YES), the processing of the processor 12 returns to step St21. On the other hand, if the processor 12A determines that the movement operation of the end effector EF1 and the camera CAM1 by the actuator ACT1 is not continuing (step St30, NO), the processing of the processor 12A shown in FIG. 6 ends.

[0091] At the time of step St28 or step St29, the processor 12A may visually notify the user of the matching result at the current position of the camera CAM1 by displaying the matching result screen WD2 shown in FIG. 7 on the display 30.

[0092] 7 is a diagram showing an example of a matching result screen WD2 displayed on a display in embodiment 2. The matching result screen WD2 is generated by, for example, the control unit 126 of the processor 12A. The matching result screen WD2 has a travel route map display area SD1A, a matching result display area SD4, a registered template display area SD3A, and a use button BT3.

[0093] The movement path map display area SD1A displays a movement path map MP2 showing the movement path (movement history) of the pair of end effector EF1 and camera CAM1. Positions p1, p2, p3, p4, p5, p7, p8, p9, p10, p11, p12, p13, p14, and p15 indicate positions where template registration was previously performed when the pair of end effector EF1 and camera CAM1 was located. Position p6 is the current position on the movement path RL1 of the end effector EF1 and camera CAM1, and is the position where the end effector EF1 and camera CAM1 currently exist (i.e., the current position) when the matching result screen WD2 of FIG. 7 was generated. Coordinates indicating the three-dimensional position of each of positions p1 to p5 and p7 to p15 are input to the image processing device 10 from the template registration device 40 in which template data (see above) is registered. On the other hand, the coordinates indicating the current position p6 are input to the image processing device 10 from the actuator ACT1.

[0094] The matching result display area SD4 shows the matching result between a captured image IMG6 of an object with a product number (e.g., "XX38YZ1X") captured by camera CAM1 at its current position (e.g., position p6) and a template TPL5 used in template matching (e.g., the template corresponding to the position immediately preceding the current position, p5). For example, the matching result display area SD4 shows that, of the 13 feature points extracted from captured image IMG6, nine feature points matched (i.e., were matched) in template matching using template TPL5. In other words, the matching rate is 9 / 13 = 69.2%. Therefore, by viewing this matching result display area SD4, the user can visually determine which feature points match between the captured image IMG6 captured at the current position of camera CAM1 and the template TPL5 used in template matching, and to what extent.

[0095] The registered template display area SD3A displays template registration images TPL1, TPL2, ..., TPL7, TPL8, TPL9, TPL10, ... when templates were registered at past registration positions (positions p1 to p5, p7 to p15 in the example of FIG. 4). Each of the template registration images TPL1, TPL2, ..., TPL7, TPL8, TPL9, TPL10, ... displays the corresponding template IMG1, IMG2, ..., IMG7, IMG8, ​​IMG9, IMG10, ... in association with the position of the camera CAM1 that captured the template (that is, the three-dimensional coordinates of the corresponding registration position). In the example of FIG. 7, there are a template registration image TPL1 including a template IMG1 in which a captured image captured by camera CAM1 at three-dimensional coordinate P1Z (0,0,0) corresponding to position p1 is registered, a template registration image TPL2 including a template IMG2 in which a captured image captured by camera CAM1 at three-dimensional coordinate P2Z (1,0,0) corresponding to position p2 is registered, ..., a template registration image TPL7 including a template IMG7 in which a captured image captured by camera CAM1 at three-dimensional coordinate P7Z (1,0,1) corresponding to position p7 is registered, and a template registration image TPL8 including a template IMG9 in which a captured image captured by camera CAM1 at three-dimensional coordinate P7Z (1,0,1) corresponding to position p8 is registered. 7 shows a template registration image TPL8 including a template IMG8 in which a captured image captured by camera CAM1 at three-dimensional coordinates P8Z (0,0,1) corresponding to position p8 is registered, a template registration image TPL9 including a template IMG9 in which a captured image captured by camera CAM1 at three-dimensional coordinates P9Z (0,0,2) corresponding to position p9 is registered, a template registration image TPL2 including a template IMG10 in which a captured image captured by camera CAM1 at three-dimensional coordinates P10Z (1,0,2) corresponding to position p10 is registered, etc. Although not shown in FIG. 7, the user can display other registered template registration images by appropriately dragging and scrolling the scroll bars SCB1 and SCB2.

[0096] The use button BT3 is a button that is pressed when a template registration image selected by a user operation from among the template registration images displayed in the registered template display area SD3A is to be used for template matching. In other words, the processor 12 of the image processing device 10 can use the selected template image for template matching by detecting that the use button BT3 has been pressed after one of the template images has been selected by a user operation.

[0097] As described above, in the picking system 100A according to the second embodiment, the image processing device 10A includes a communication unit (e.g., communication interface 11) that communicates with the template registration device 40, which stores multiple templates of the object OB1 in association with position information of the camera CAM1 that can capture and move the object OB1; an acquisition unit (e.g., communication interface 11) that acquires the position information of the camera CAM1; a template prediction unit 128 that performs a prediction process to predict a template to be used for template matching of the object OB1 from the multiple templates stored in the template registration device 40, based on the position information of the camera CAM1; and a matching unit (e.g., feature matching unit 122) that performs template matching using an input image of the object OB1 captured by the camera CAM1 and the result of the template prediction process. As a result, the image processing device 10A can achieve highly accurate template matching of the object OB1 even in a situation where the orientation of the object OB1 from the camera CAM1 changes as the camera CAM1 moves.

[0098] Furthermore, the feature matching unit 122 performs template matching using a template specified by the operation device 20. This allows the image processing device 10A to efficiently perform template matching using a template selected by the user using the operation device 20, even when performing template matching for the first time, for example.

[0099] Furthermore, the template prediction unit 128 acquires position information of the camera CAM1 every time the camera CAM1 moves, and performs template prediction processing using the acquired position information. This allows the image processing device 10A to predict and select an appropriate template to be used for template matching every time the camera CAM1 moves.

[0100] The image processing device 10A also includes a feature extraction unit 121 that extracts features of the object OB1 from the input image. The feature matching unit 122 performs template matching, which is a matching process between the features of the object OB1 that appear in the result of the template prediction process and the extracted results of the features of the object OB1 that appear in the input image. This allows the image processing device 10A to predict an appropriate template to be used for template matching by comparing the features of the object OB1 that appear in the input image and the template.

[0101] The image processing device 10A further includes a template update determination unit 129 that determines whether or not to update the template used in template matching based on either the movement direction based on the position information of the camera CAM1 or the result of template matching. This allows the image processing device 10A to appropriately determine whether or not the template used in template matching needs to be updated (in other words, switched to another template).

[0102] Furthermore, the feature matching unit 122 performs template matching using the same template while the camera CAM1 is moving over a fixed distance. As a result, the image processing device 10A can assume that the appearance of the object OB1 as seen from the camera CAM1 (in other words, its posture) remains almost unchanged while the camera CAM1 is moving over a fixed distance, such as a very small distance, and can use the currently used template as is, thereby suppressing the need to determine whether or not to update the template or the increase in the processing load required for the update.

[0103] The image processing device 10A also includes a feature extraction unit 121 that extracts features of the object OB1 from the input image, and a control unit 126 that displays the result of the template prediction process and the input image in a contrastive manner on the display 30. The control unit 126 displays identical feature points between the features of the object OB1 extracted in the result of the template prediction process and the features of the object OB1 extracted in the input image by connecting them with lines in an identifiable manner. This allows the user to visually determine the matching status (e.g., the number and percentage of matching feature points) of the feature points displayed in a contrastive manner on the display 30, and visually determine which feature points match and to what extent between the image captured at the current position of the camera CAM1 and the template used in template matching.

[0104] The image processing device 10A further includes a control unit 126 that displays a movement path map MP2 of the camera CAM1 on the display 30. This allows the user to visually determine the path along which the end effector EF1 is moving to pick up the target object OB1 in the picking system 100A.

[0105] The image processing device 10A also includes a control unit 126 that associates position information of the camera CAM1, which indicates the registration position of the template, with each template registered in the template registration device 40 and displays the associated template on the display 30. This allows the user to visually check not only a list of previously registered templates but also the posture of the object OB1 reflected in each template.

[0106] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0107] The present disclosure is useful as a template registration device, a template registration method, and a template registration system that register high-precision templates of an object that can be used for template matching even in a situation where the orientation of the object from an image capturing device changes as the image capturing device moves. [Explanation of symbols]

[0108] 10, 10A Image processing device 11 Communication Interface 12, 12A processor 13. Memory 20 Operation Device 30 Display 40 Template Registration Devices 100, 100A Picking System 121 Feature Extraction Unit 122 Feature Matching Unit 123 MoS calculation section 124 MaS calculation section 125 Template registration decision unit 126 Control Section 127 Position fitting part 128 Template Prediction Unit 129 Template update decision unit ACT1 Actuator CAM1 camera

Claims

1. a communication unit that communicates with a storage unit that stores a plurality of templates of an object in association with position information of an imaging device that can image and move the object; an acquisition unit that acquires position information of the imaging device; a prediction unit that performs a prediction process to predict a template to be used for template matching of the object from among the plurality of templates stored in the storage unit, based on position information of the imaging device; a matching unit that performs the template matching using an input image of the object captured by the imaging device and a result of the prediction process of the template, Template matching device.

2. the matching unit performs the template matching using a template designated by an operation device.

2. The template matching device according to claim 1.

3. the prediction unit acquires position information of the imaging device every time the imaging device moves, and performs the prediction process of the template using the acquired position information.

2. The template matching device according to claim 1.

4. a feature extraction unit that extracts feature points of the object from the input image, the matching unit performs template matching, which is a matching process between a result of the prediction process of the template or a feature of the object shown in a template designated by an operation device and an extraction result of the feature of the object shown in the input image; 2. The template matching device according to claim 1.

5. an update determination unit that determines whether to update the template used in the template matching based on either a moving direction based on position information of the imaging device or a result of the template matching, 5. The template matching device according to claim 4.

6. the matching unit performs the template matching using the same template while the imaging device is moving within a certain distance.

2. The template matching device according to claim 1.

7. a feature extraction unit that extracts features of the object from the input image; a control unit that displays a result of the template prediction process and the input image on a display in a comparative manner, the control unit displays, by connecting with lines, identical feature points of the feature of the object extracted in the result of the prediction process of the template and the feature of the object extracted in the input image in a manner that allows them to be distinguished.

2. The template matching device according to claim 1.

8. a control unit that displays a movement route map of the imaging device on a display; 2. The template matching device according to claim 1.

9. a control unit that displays, on a display, for each of the templates registered in the storage unit, position information of the imaging device that indicates a registration position of the template in association with the template.

2. The template matching device according to claim 1.

10. A template matching method executed by a template matching device, comprising: a step of communicating with a storage unit that stores a plurality of position information of an imaging device that can image and move an object and templates of the object in association with each other; acquiring location information of the imaging device; performing a prediction process for predicting a template to be used for template matching of the object from among the plurality of templates stored in the storage unit based on position information of the imaging device; performing the template matching using an input image of the object captured by the imaging device and a result of the template prediction process; Template matching method.

11. an imaging device capable of imaging and moving an object; a template matching device communicably connected to the imaging device, The template matching device a communication unit that communicates with a storage unit that stores a plurality of position information of the imaging device and templates of the object in association with each other; an acquisition unit that acquires position information of the imaging device; a prediction unit that performs a prediction process to predict a template to be used for template matching of the object from among the plurality of templates stored in the storage unit, based on position information of the imaging device; a matching unit that performs the template matching using an input image of the object captured by the imaging device and a result of the template prediction process, Template matching system.

12. the imaging device is movable in three dimensions by an actuator and is fixed to a movable section to which an end effector capable of picking up the object is fixed, and is movable; The template matching system of claim 11.

Images