Image processing method and image processing device

The image processing method and device address inefficiencies in object recognition by predicting object posture changes, ensuring accurate template matching and maintaining production efficiency in factory environments.

JP2024158428APending Publication Date: 2024-11-08PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023073618
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing object recognition systems in factories require multiple stages of template matching, increasing processing load and reducing efficiency when the camera and end effector move, leading to unreliable template matching due to changes in the object's posture.

Method used

An image processing method and device that predicts the posture of an object using multiple template generations based on detected position and orientation changes, allowing for efficient template matching without stopping conveyor operations.

Benefits of technology

Enables highly accurate template matching even when the imaging device moves, maintaining production efficiency by predicting and adapting to changes in object posture.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158428000001_ABST
    Figure 2024158428000001_ABST
Patent Text Reader

Abstract

To accomplish a highly precise template matching on an object even under a situation in which the attitude of the object from an imaging device is variable due to the displacement of the imaging device.SOLUTION: An image processing method is performed by an image processing device communicable with a camera which can be displaced and which can pick up the image of an object, the method including steps of: obtaining the picked-up image of the object; detecting the position of the object or the attitude thereof from the picked-up image; executing a first process that generates a template for the object; executing a template matching based on the picked-up image and on the template for the object during the execution of the first process; and executing, by multiple times, a second process that obtains information relating to the displacement of the object. The first process predicts the attitude of the object based on the detected position of the object or attitude thereof and on information relating to the displacement of the object which is obtained in the second process by multiple times, and generates a template for the object corresponding to the predicted attitude of the object.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an image processing method and an image processing device. [Background technology]

[0002] In a production process in a factory, it is sometimes necessary to determine whether a part to be picked by an end effector such as a robot hand is a correct part (for example, a part to be used in the production of an industrial product). When making such a determination, it is required to perform the determination process as quickly as possible so as not to reduce the takt time of the production process. As a conventional determination process, for example, a template matching method is known in which a template (for example, an image) of a part prepared in advance is compared with an image of the part captured by a camera installed in the factory to perform a matching process.

[0003] Patent Document 1 discloses a template creation device that creates a set of templates used in an object recognition device that recognizes objects by template matching. The template creation device acquires multiple templates from multiple images of an object in different postures, calculates the similarity of image features between two templates selected from the multiple templates, and performs clustering to divide the multiple templates into multiple groups based on the similarity. The template creation device integrates all templates in each of the multiple groups into one integrated template, and generates a template set having an integrated template for each group. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-207147 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, the object recognition device performs a hierarchical search by creating a hierarchical template set, performing rough recognition using a low-resolution template set, and using the result to perform detailed recognition using a high-resolution template set. However, it is necessary to perform matching processing in at least two stages, such as recognition processing using a low-resolution template set and recognition processing using a high-resolution template set, and this inevitably increases the processing load of the object recognition device.

[0006] Furthermore, when the end effector and the camera are moved to capture an image of a part to be picked by the end effector in the above-mentioned production process in a factory in order to determine whether the part to be picked by the end effector is the correct part, the following problem occurs when applying the technology of Patent Document 1. Specifically, if the camera moves with the movement of the end effector, the way the part is viewed by the camera (in other words, the posture of the part) changes with the change in the position of the end effector. For this reason, if the position of the end effector (the position of the camera, so to speak) is not taken into consideration during template matching, efficient template matching cannot be performed even if a template set generated in advance is used, and the reliability of template matching does not improve.

[0007] The present disclosure has been devised in consideration of the conventional circumstances, and aims to provide an image processing method and an image processing device that realize highly accurate template matching of an object even when the posture of the object relative to an imaging device changes as the imaging device moves. [Means for solving the problem]

[0008] The present disclosure provides an image processing method performed by an image processing device that is mobile and capable of communicating with a camera capable of capturing an image of an object, the image processing method comprising: acquiring an image of the object; detecting a position or posture of the object from the captured image; executing a first process to generate a template of the object; and, while executing the first process, executing template matching based on the captured image and the template of the object; and executing a second process multiple times to acquire information regarding a movement of the object, the first process predicting the posture of the object based on the detected position or posture of the object and information regarding the movement of the object acquired multiple times by the second process; and generating a template of the object corresponding to the predicted posture of the object.

[0009] The present disclosure also provides an image processing device comprising: an acquisition unit that acquires an image of an object captured by a movable camera capable of capturing an image of the object; a first processing unit that detects a position or posture of the object from the captured image and generates a template of the object; and a second processing unit that performs template matching based on the captured image and the object template multiple times while the first processing unit is generating the template of the object, and acquires information regarding movement of the object, wherein the first processing unit predicts the posture of the object based on the detected position or posture of the object and the information regarding the movement of the object acquired multiple times by the second processing unit, and generates the template of the object corresponding to the predicted posture of the object. Effect of the Invention

[0010] According to the present disclosure, highly accurate template matching of an object can be achieved even in a situation where the orientation of the object from an imaging device changes as the imaging device moves. [Brief description of the drawings]

[0011] [Figure 1] A diagram explaining an example of the configuration of a picking system [Diagram 2]Block diagram showing an example of the internal configuration of a picking system [Diagram 3] FIG. 1 is a functional block diagram illustrating functions of a first processing unit and a second processing unit according to a first embodiment. [Figure 4] FIG. 1 is a diagram for explaining an example of an overall operation procedure of a picking system according to the first embodiment. [Diagram 5] 11 is a flowchart showing a second example of a processing procedure of the image processing device according to the first embodiment. [Figure 6] 1 is a flowchart showing a first example of a processing procedure of the image processing device according to the first embodiment. [Figure 7] FIG. 13 is a diagram showing an example of a template display screen. [Figure 8] FIG. 1 is a diagram for explaining an example of an overall operation procedure of a picking system according to a first modification of the first embodiment. [Figure 9] A flowchart showing an example of an operation procedure of a first processing unit of an image processing device according to a second modification of the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Background to this disclosure) Conventionally, an object recognition device has been disclosed that includes a template creation device for objects on a belt conveyor using images captured by a camera installed at a fixed location that cannot be moved relative to a production line, etc. In addition, when actually performing matching processing, the object recognition device uses a template set generated based on an integrated template for each group as is.

[0013] However, when the end effector and the camera are moved to capture an image of the part to be picked by the camera in order to determine whether the part to be picked by the end effector is the correct part in the above-mentioned production process in the factory, the following problem occurs when attempting to apply the technology of Patent Document 1 (JP Patent Publication 2016-207147 A). Specifically, if the camera moves with the movement of the end effector, the way the part is viewed by the camera (in other words, the posture of the part) changes with the change in the position of the end effector. For this reason, if the position of the end effector (the position of the camera, so to speak) is not taken into consideration during template matching, efficient template matching cannot be performed even if a template set generated in advance is used, and the reliability of template matching is not improved.

[0014] In order to solve this problem, there is a method of stopping or slowing down the belt conveyor in accordance with the matching process speed, but when such a method is adopted, the part picking efficiency (in other words, production efficiency) decreases. Therefore, there has been a demand for an object recognition device that can realize part matching and part picking without stopping the operation of the belt conveyor and end effector.

[0015] Therefore, in the following embodiments, examples of an image processing method and an image processing device are described that register a highly accurate template of an object that can be used for template matching even in a situation where the orientation of the object from the imaging device changes as the imaging device moves.

[0016] Hereinafter, with reference to the attached drawings as appropriate, each embodiment specifically disclosing the image processing method and image processing device according to the present disclosure will be described in detail. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters and duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the attached drawings and the following explanation are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0017] (Embodiment 1) In the first embodiment, for example, in a production process in a factory, when determining whether a part to be picked by an end effector such as a robot hand (for example, a part used in the production of an industrial product) is correctly recognized by template matching, an object recognition process is performed on an image captured by a camera, and a template required for template matching is predicted. A template registration device (for example, an image processing device) according to the present disclosure images an object, and registers information (see below) based on an input image of an object captured by an imaging device that can change the imaging position with respect to the object by moving, and information (see below) based on the input image of the object as a template to be used for template matching, in a storage unit, in association with position information of the imaging device and information (see below) based on the input image of the object.

[0018] Fig. 1 is a diagram for explaining a configuration example of a picking system. Fig. 2 is a block diagram showing an internal configuration example of the picking system. As shown in Fig. 2, the picking system 100 includes an actuator AC, a camera CM, an image processing device P1, a display 13, and an operation device 14. The actuator AC and the image processing device P1, the camera CM and the image processing device P1, and the image processing device P1 and the operation device 14 are connected so as to enable input / output (transmission / reception) of data signals.

[0019] The positional relationship between the camera CM and the target object Tg will be described with reference to Fig. 1. Note that the description of Fig. 1 is applicable not only to the first embodiment but also to a modification of the first embodiment described later.

[0020] In the following description, the object Tg is an object to be picked by the end effector EF of the picking system 100 installed in a factory, such as an industrial part or industrial product. If it is an industrial part, it is moved to another lane (production line) after being picked, for example, to assemble a finished product. If it is an industrial product, it is stored in a box such as a cardboard box, for example, after being picked. It goes without saying that the type of object Tg is not limited to the above-mentioned industrial parts and industrial products.

[0021] As shown in FIG. 1, the actuator AC controls the camera CM to be movable three-dimensionally, thereby changing the positional relationship between an object Tg moving on a conveyor belt, an end effector EF that picks up the object Tg, and a camera CM fixed to the end effector EF.

[0022] The actuator AC controls the end effector EF and the camera CM provided on the end effector EF so that they can be moved three-dimensionally using multiple axes. In other words, the actuator AC can recognize, maintain, and change the three-dimensional position (coordinates) of the camera CM.

[0023] The end effector EF is, for example, a robot hand provided at the tip of a robot arm arranged corresponding to the picking system 100, and approaches the target object Tg under the control of the actuator AC and picks up the target object Tg.

[0024] The camera CM is disposed near the end effector EF and moves together with the end effector EF under the control of the actuator AC to capture an image of the object Tg. The camera CM captures an image of the object Tg at a predetermined frame rate (e.g., 1000 frames per rate (hereinafter, referred to as "fps")) and transmits the captured image (an example of an input image) of the object Tg obtained each time the image is captured to the image processing device P1.

[0025] The image processing device P1 acquires a captured image of the object Tg transmitted from the camera CM. The image processing device P1 is configured by a computer capable of executing a first process (see FIG. 4 and FIG. 6) of detecting the posture of the object Tg using the captured image of the object Tg transmitted from the camera CM, observing the movement of the object Tg based on information on the movement of the object Tg obtained by the second process while detecting the posture of the object Tg, predicting the posture of the object Tg at the timing when the posture detection process of the object Tg ends based on the detected posture of the object Tg and the movement information of the object Tg, and generating a detection template corresponding to the predicted posture of the object Tg (an example of the detected posture), and a second process (see FIG. 4 and FIG. 5) of extracting a feature amount of the object Tg, acquiring position information of the object Tg by performing feature matching (in other words, template matching) based on the extracted feature amount, and constantly or periodically transmitting (feeding back) the acquired position information of the object Tg to the actuator AC.

[0026] The image processing device P1 may be, for example, a personal computer (hereinafter, referred to as "PC"), or may be a dedicated hardware device specialized for executing each of the above-mentioned first process and second process. The image processing device P1 realizes a recognition process of the position and orientation of the target object Tg picked up by the end effector EF by executing each of the above-mentioned first process and second process. The image processing device P1 includes a communication unit 10, a processor 11, a memory 12, and a 3D model database DB.

[0027] The image processing device P1 accepts a user operation and, based on the user operation, generates a template display screen SC (see Figure 7) including a template TP1 (see Figure 7) of the object Tg viewed from the position and posture of the icon PP1 described later, a prediction template TP2 (see Figure 7) of the object Tg corresponding to the predicted posture of the object Tg predicted by the first processing, and a detection template TP3 (see Figure 7) of the object Tg corresponding to the posture of the object Tg detected by the first processing, and displays it on the display 13.

[0028] The communication unit 10 (an example of an acquisition unit) is connected to the actuator AC, the camera CM, the display 13, and the operation device 14 so as to be able to perform data communication with each of them, and executes transmission and reception of data. The communication unit 10 outputs the captured image transmitted from the camera CM and the control command transmitted from the operation device 14 to the processor 11. The communication unit 10 transmits the template display screen SC (see FIG. 7) output from the processor 11 to the display 13.

[0029] The processor 11 is configured using, for example, a Central Processing Unit (CPU) or a Field Programmable Gate Array (FPGA), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 refers to the programs and data stored in the memory 12, and executes the programs to realize the functions of the first processing unit 110 and the second processing unit 120.

[0030] The first processing unit 110 executes a detection process of the object Tg and executes a first process (see FIG. 4 and FIG. 6) for generating a prediction template TP2 of the object Tg. The first process is an advanced image process using deep learning, and is a process that requires a longer time (for example, 17 ms) than the time required for executing the second process. Here, the position and posture of the object Tg continue to change while the first process is being executed. Therefore, in order to maintain the validity of the detection template TP3 output at the timing when the first process is completed, the first processing unit 110 predicts template candidates of the object Tg using the results of the second process executed by the second processing unit 120, which is executed multiple times while the first process is being executed. As a result, the first processing unit 110 can track the posture change of the object Tg that changes during the execution of the first process in real time, and predict template candidates that are more suitable for feature matching executed by the second processing unit 120.

[0031] The second processing unit 120 executes a second process (see Figs. 4 and 5) in which feature matching is performed using the prediction template TP2 obtained by the first process and a captured image captured by the camera CM, and the second process estimates position information of the object Tg and transmits the estimated position information of the object Tg to the actuator AC. The second process is a simple image process using feature matching, and requires a shorter time (e.g., 1 ms) than the time required to execute the first process. The second processing unit 120 executes the second process multiple times while the first processing unit 110 executes the first process once.

[0032] The memory 12 has, for example, a Random Access Memory (RAM) as a work memory used when the processor 11 executes each process, and a Read Only Memory (ROM) for storing programs and data that define the operation of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The ROM has written therein programs that define the operation of the processor 11.

[0033] The 3D model database DB (an example of a database) is, for example, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The 3D model database DB stores (registers) for each object Tg, data of a 3D model of at least one object Tg to be picked, and information about the object Tg (for example, the name and identification number of each object).

[0034] The display 13 is a device that outputs (displays) the template display screen SC (see FIG. 7) generated by the image processing device P1, and is configured, for example, by a Liquid Crystal Display (LCD) or an organic electroluminescence (EL) device.

[0035] The operation device 14 is an interface that detects input of a user operation and is configured with, for example, a mouse, a keyboard, or a touch panel. When the operation device 14 receives a user operation, it generates an electric signal based on the user operation and transmits the electric signal to the image processing device P1.

[0036] Next, functions realized by each of the first processing unit 110 and the second processing unit 120 will be described with reference to Fig. 3. Fig. 3 is a functional block diagram illustrating the functions of the first processing unit 110 and the second processing unit 120 in the first embodiment.

[0037] The first processing unit 110 includes an object detection unit 111, a 3D model selection unit 112, a first time prediction unit 113, and a 3D model synthesis unit 117.

[0038] The object detection unit 111 uses deep learning to execute image recognition processing on the captured image transmitted from the camera CM, and detects an object (target object Tg) from the captured image. The object detection unit 111 outputs information on the detected object (target object Tg) to the 3D model selection unit 112 and the first time prediction unit 113. Note that the deep learning used by the object detection unit 111 may be any learning method suitable for detecting the target object Tg, such as a Convolutional Neural Network (CNN).

[0039] The 3D model selection unit 112 selects a 3D model MD corresponding to information of an object (object Tg) specified by a user operation from among at least one 3D model of an object Tg registered in the 3D model database DB. The 3D model selection unit 112 outputs the selected 3D model MD of the object Tg to each of the 3D matching unit 114 and the 3D model synthesis unit 117.

[0040] The first time prediction unit 113 tracks (observes) changes in the position and posture of the object Tg based on the motion information of the object Tg obtained by the second process while performing the detection process of the object Tg by the first process, and generates a prediction template TP2 corresponding to the posture of the object Tg at the timing when the first process ends. The first time prediction unit 113 includes a 3D matching unit 114, a template prediction unit 115, and a prediction model update unit 116.

[0041] The 3D matching unit 114 executes 3D matching for matching, in a three-dimensional space, information on the object (target object Tg) output from the object detection unit 111 with the 3D model MD of the target object Tg output from the 3D model selection unit 112, and recognizes the posture of the target object Tg appearing in the captured image captured by the camera CM (hereinafter, referred to as the "detected posture"). The 3D matching unit 114 associates information on the detected posture of the target object Tg with the 3D model MD of the target object Tg used in the 3D matching, and outputs the information to the template prediction unit 115 and the 3D model synthesis unit 117, respectively.

[0042] The template prediction unit 115 generates a prediction template TP2 corresponding to the posture of the object Tg at the time when the first process ends, based on information on the detected posture of the object Tg and the 3D model MD of the object Tg transmitted from the 3D matching unit 114, and the prediction model output from the prediction model update unit 116. Note that the prediction model here is a prediction model in which the posture of the object Tg is predicted, and is a mathematical model based on information on the movement of the object Tg obtained by the second process executed multiple times.

[0043] Based on the prediction result, the template prediction unit 115 generates a 2D image (hereinafter, referred to as a "prediction template") obtained when the 3D model MD is viewed from the predicted posture (angle) of the object Tg, which is the posture of the camera CM when the captured image to be acquired next is captured. The template prediction unit 115 associates information about the posture of the object Tg with the generated prediction template TP2 of the object Tg (see FIG. 7), and outputs the information to the 3D model synthesis unit 117 and the template update unit 121.

[0044] The prediction model update unit 116 updates the prediction model for predicting a change in the posture of the object Tg that changes while the detection process for the object Tg is being performed by the first process, based on the feature matching result output from the feature matching unit 123 and the movement of the object Tg output from the position fitting unit 124. The prediction model update unit 116 outputs the updated prediction model to the template prediction unit 115.

[0045] The 3D model synthesis unit 117 acquires the detected orientation of the object Tg output from the 3D matching unit 114 and the detection template TP3 (see FIG. 7), and generates an icon PP3 indicating the angle at which the object Tg in the captured image was captured based on the detected orientation of the object Tg. The 3D model synthesis unit 117 acquires the predicted orientation of the object Tg output from the template prediction unit 115 and the prediction template TP2, and generates an icon PP2 indicating the angle of the object Tg to be captured in the next captured image based on the predicted orientation of the object Tg.

[0046] The 3D model synthesis unit 117 generates a template display screen SC based on the 3D model MD of the object Tg, the prediction template TP2, the detection template TP3, and each of the icons PP1, PP2, and PP3, and transmits it to the display 13.

[0047] The second processing unit 120 includes a template updating unit 121 , a feature extracting unit 122 , a feature matching unit 123 , a position fitting unit 124 , a second time prediction unit 125 , and a control unit 126 .

[0048] The template update unit 121 acquires the prediction template TP2 output from the template prediction unit 115 of the first processing unit 110, and updates the template (2D data) of the object Tg used for feature matching to the acquired prediction template TP2.

[0049] The feature extraction unit 122 extracts a feature amount of the object Tg from the captured image transmitted from the camera CM. The feature extraction unit 122 outputs the extracted feature amount of the object Tg to the feature matching unit 123.

[0050] The feature matching unit 123 matches the feature amount of the object Tg included in the prediction template TP2 output from the template update unit 121 with the feature amount of the object Tg output from the feature extraction unit 122. The feature matching unit 123 outputs the matching result to each of the prediction model update unit and the position fitting unit 124.

[0051] The position fitting unit 124 acquires the matching result output from the feature matching unit 123. The position fitting unit 124 fits the position information of the object Tg appearing in the captured image based on the result of the feature matching.

[0052] The position fitting unit 124 outputs the position information of the target object Tg after the position fitting to the prediction model updating unit 116 and the second time prediction unit 125, respectively.

[0053] The second time prediction unit 125 predicts the movement of the object Tg during execution of the second process based on the position information of the object Tg output from the position fitting unit 124, and predicts the position of the object Tg at the time when the second process ends. The second time prediction unit 125 outputs information on the predicted position of the object Tg to the control unit 126.

[0054] The control unit 126 outputs the information on the predicted position of the target object Tg output from the second time prediction unit 125 to the actuator AC.

[0055] Next, an overall operation procedure of the picking system 100 will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining an example of an overall operation procedure of the picking system 100 according to the first embodiment.

[0056] The overall operation procedure example of the picking system 100 shown in Fig. 4 is an example, and is not limited to this. Fig. 4 shows an example in which the first process is executed once and the second process is executed N times (N: an integer of 3 or more) in picking one target Tg in order to make the relationship between the first process and the second process easier to understand, but the number of times the first process and the second process are executed is not limited to this. It goes without saying that the picking system 100 may execute the first process multiple times in picking one target Tg.

[0057] The actuator AC picks up the object Tg conveyed on the belt conveyor while capturing images of the object Tg by the camera CM at a predetermined frame rate (for example, 1000 fps). The actuator AC shown in Fig. 4 shows a part of the picking process of the object Tg (time t11 to time t1N), and is repeatedly executed until the object Tg is picked up, for example.

[0058] The image processing device P1 acquires a captured image of the object Tg captured by the camera CM at a predetermined frame rate, and executes a second process (step St100) on the acquired captured image.

[0059] In parallel with the second process, the image processing device P1 executes the first process (step St200) based on the captured image captured by the camera CM, the feature matching results (in other words, matching tendency) obtained by the second process executed multiple times during the first process, and the movement amount (in other words, movement information) of the object Tg. The image processing device P1 feeds back the prediction template TP2 (see FIG. 7), which is a template candidate for the object Tg obtained by the first process, to the second process.

[0060] In the example shown in FIG. 4, the camera CM captures an object Tg at time t11 and transmits the captured image Img11 to the image processing device P1. The image processing device P1 acquires the first captured image Img11 (image data) transmitted from the camera CM and executes the first process and the second process using the first captured image Img11. The image processing device P1 outputs the matching tendency of the object Tg and the movement information of the object Tg obtained by the second process to the first processing unit 110, and transmits information of the predicted position (x1, y1, z1) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired three-dimensional predicted position (x1, y1, z1) of the object Tg.

[0061] At time t12, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img12 to the image processing device P1. The image processing device P1 acquires the second captured image Img12 (image data) transmitted from the camera CM and executes a second process using the second captured image Img12. The image processing device P1 outputs the matching tendency of the object Tg and the movement information of the object Tg obtained by the second process to the first processing unit 110, and transmits information of the predicted position (x2, y2, z2) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position (x2, y2, z2) of the object Tg.

[0062] At time t13, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img13 to the image processing device P1. The image processing device P1 acquires the third captured image Img13 (image data) transmitted from the camera CM and executes the second process using the third captured image Img13. The image processing device P1 outputs the matching tendency of the object Tg and the movement information of the object Tg obtained by the second process to the first processing unit 110, and transmits information on the predicted position (not shown) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position of the object Tg.

[0063] At time t1(N-2), the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img1(N-2) to the image processing device P1. The image processing device P1 acquires the (N-2)th captured image Img1(N-2) (image data) transmitted from the camera CM, and executes the second process using the (N-2)th captured image Img1(N-2). The image processing device P1 outputs the matching tendency of the object Tg and the movement information of the object Tg obtained by the second process to the first processing unit 110, and transmits information of the predicted position (not shown) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position of the object Tg.

[0064] At time t1(N-1), the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img1(N-1) to the image processing device P1. The image processing device P1 acquires the (N-1)th captured image Img1(N-1) (image data) transmitted from the camera CM, and executes the second process using the (N-1)th captured image Img1(N-1). The image processing device P1 outputs the matching tendency of the object Tg and the movement information of the object Tg obtained by the second process to the first processing unit 110, and transmits information of the predicted position (not shown) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position of the object Tg.

[0065] At time t1N, the camera CM captures an object Tg. The camera CM transmits the captured image Img1N to the image processing device P1. The image processing device P1 acquires the Nth captured image Img1N (image data) transmitted from the camera CM and executes the second process using the Nth captured image Img1N. The image processing device P1 outputs the matching tendency of the object Tg and the movement information of the object Tg obtained by the second process to the first processing device 110, and transmits information on the predicted position (xN, yN, zN) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position (xN, yN, zN) of the object Tg to pick up the object Tg. The image processing device P1 feeds back the template candidate (prediction template TP2) of the object Tg obtained by the first process at time t1(N+1) to the second processing device 120 and updates the template candidate (prediction template TP2).

[0066] Although not shown in the example of FIG. 4, in the second process executed subsequently, the image processing device P1 performs feature matching using the latest obtained prediction template TP2 until a new prediction template TP2 is fed back again by the first process.

[0067] Next, the second process of the image processing device P1 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the second process procedure (step St100) of the image processing device P1 according to the first embodiment.

[0068] The second processing unit 120 sets an update flag for updating the template of the target object Tg to "1" based on a control command notifying the start of the picking process acquired via the operation device 14 (St11). The second processing unit 120 acquires a captured image transmitted from the camera CM (St12).

[0069] The second processing unit 120 determines whether the currently set update flag is "1" or not (St13).

[0070] When the second processing unit 120 determines in the process of step St13 that the currently set update flag is "1" (St13, YES), it updates the template (template TP1 or prediction template TP2) of the object Tg used for feature matching (St14). The second processing unit 120 outputs the captured image of the object Tg for generating a template candidate (prediction template TP2) to the first processing unit 110 (St15), and sets the update flag to "0" (St16).

[0071] On the other hand, if the second processing unit 120 determines in the processing of step St13 that the currently set update flag is not "1" (St13, NO), it determines whether or not there is a template (prediction template TP2) of the object Tg to be used for feature matching (St17).

[0072] When it is determined in the process of step St17 that there is a template (prediction template TP2) of the object Tg used for feature matching (St17, YES), the second processing unit 120 extracts a feature amount from the captured image (St18).

[0073] On the other hand, if the second processing unit 120 determines in the processing of step St17 that there is no template (prediction template TP2) of the object Tg to be used for feature matching (St17, NO), it returns to the processing of step St13 and waits until a template (template TP1 or prediction template TP2) of the object Tg is fed back from the first processing unit 110.

[0074] The second processing unit 120 performs feature matching between the feature amount extracted from the captured image (i.e., the feature amount of the captured image) and the feature amount of the object Tg based on the template (i.e., the feature amount of the template). The second processing unit 120 fits the position of the object Tg appearing in the captured image captured by the camera CM based on the matching result (St19).

[0075] The second processing unit 120 calculates the amount of movement of the object Tg between the captured image used for feature matching (i.e., the latest captured image) and the captured image captured immediately before this captured image. Based on the calculated amount of movement, the second processing unit 120 predicts the amount of movement of the object Tg during execution of the second process, predicts the position of the object Tg at the timing when the second process ends, and obtains the predicted position of the object Tg (St20). The second processing unit 120 transmits the predicted position of the object Tg to the actuator AC (St21).

[0076] The second processing unit 120 outputs the matching tendency, which is the result of feature matching, and the movement information (movement information) of the object Tg to the first processing unit 110 (St22).

[0077] The second processing unit 120 determines whether or not the picking process of the object Tg has been completed based on a control command notifying the completion of the picking process of the object Tg by the actuator AC (St23).

[0078] When it is determined in the process of step St23 that the picking process of the target object Tg has ended (St23, YES), the second processing unit 120 ends the second process (step St100) shown in FIG.

[0079] On the other hand, when the second processing unit 120 determines in the process of step St23 that the picking process of the target object Tg has not been completed (St23, NO), the process returns to the process of step St11.

[0080] As described above, the image processing device P1 in the first embodiment can predict the amount of movement of the object Tg during the execution of the second process based on the captured images captured at a high frame rate by the second process, and can predict the position of the object Tg at the timing when the second process ends. This allows the image processing device P1 to support the actuator AC in tracking the object Tg in real time.

[0081] Next, the first process of the image processing device P1 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the first process procedure (step St200) of the image processing device P1 according to the first embodiment.

[0082] The first processing unit 110 acquires the captured image output from the second processing unit 120 (St31). The first processing unit 110 detects the object Tg from the acquired captured image by using an advanced image processing technique such as Deep Learning (St32).

[0083] Based on the detected object Tg, the first processing unit 110 selects a 3D model MD corresponding to the object Tg from among the 3D models stored in the 3D model database DB. The first processing unit 110 performs 3D matching between the selected 3D model MD and the detected object Tg to obtain a detected orientation of the object Tg appearing in the captured image (in other words, the imaging orientation in which the object Tg is captured) (St33). This allows the first processing unit 110 to obtain a detected orientation for generating the detection template TP3 (see FIG. 7). The first processing unit 110 updates a prediction model for predicting the orientation of the object Tg (in other words, the imaging orientation) based on the feature matching result and the movement information of the object Tg acquired from the second processing unit 120 (St34).

[0084] The first processing unit 110 predicts a template candidate corresponding to the posture of the object Tg at the timing (time) when the first processing is completed based on a prediction model for predicting the updated posture of the object Tg (in other words, the imaging posture) and the detected posture of the object Tg (St35), and feeds back (outputs) the predicted template candidate (prediction template TP2) to the second processing unit 120 (St36).

[0085] The first processing unit 110 sets an update flag for updating the template of the object Tg to "1" (St37).

[0086] The first processing unit 110 determines whether or not the picking process of the object Tg has been completed, based on a control command notifying the completion of the picking process of the object Tg by the actuator AC (St38).

[0087] When it is determined in the process of step St38 that the picking process of the target object Tg has ended (St38, YES), the first processing unit 110 ends the first process (step St200) shown in FIG.

[0088] On the other hand, when the first processing unit 110 determines in the process of step St38 that the picking process of the target object Tg has not been completed (St38, NO), the process returns to the process of step St31.

[0089] As described above, the image processing device P1 in the first embodiment can select with higher accuracy the 3D model MD of the object Tg for generating the prediction template TP2 used in the second process by performing a more advanced image recognition process in the first process. In addition, the image processing device P1 can track the change in posture of the object Tg that changes during the first process based on the feature matching result and the motion information of the object Tg output from the second processing unit 120, which has a short processing time, and generate a template (prediction template TP2) of the object Tg that is closer to the posture of the actual object Tg. As a result, the image processing device P1 can improve the feature matching accuracy of the second processing unit 120 and can improve the tracking accuracy of tracking the object Tg.

[0090] Next, the template display screen will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the template display screen SC. Note that the template display screen SC shown in Fig. 7 is only an example and is not limiting.

[0091] The first processing unit 110 generates a template display screen SC based on a result of the first processing, and transmits the generated template display screen SC to the display 13 for display. The template display screen SC includes a first display area AR1, a second display area AR2, a third display area AR3, a fourth display area AR4, a fifth display area AR5, and a registration button BT.

[0092] The first display area AR1 includes a 3D model MD of an object Tg obtained by image processing in the first process or selected by user operation in a variant of embodiment 1 described below, and an XYZ coordinate system set in the data of the 3D model MD of the object Tg.

[0093] In addition, the first display area AR1 includes an icon PP1 indicating the imaging angle of the camera CM corresponding to the template TP1, an icon PP2 indicating the imaging angle of the camera CM corresponding to the prediction template TP2 (an example of a first imaging position), and an icon PP3 indicating the imaging angle of the camera CM corresponding to the detection template TP3 (an example of a second imaging position).

[0094] The icon PP1 can receive a user operation via the operation device 14. When the position of the icon PP1 is changed by a user operation, the image processing device P1 generates a template (2D) of the 3D model MD of the object Tg viewed from the position (angle) of the icon PP1. The image processing device P1 generates a template display screen SC in which the generated template (2D) of the 3D model MD is displayed as a template TP1 in the third display area AR3, and transmits the template to the display 13 for display.

[0095] The icon PP2 is the position (angle) of the camera CM capable of capturing an image of the template (2D) of the 3D model MD of the prediction template TP2, and indicates the imaging position of the object Tg to be imaged next. When the image processing device P1 predicts the predicted attitude of the object Tg by the first process based on the predicted position of the object Tg obtained by the second process, the image processing device P1 updates the position of the icon PP2 and the prediction template TP2 displayed in the fourth display area AR4 based on the predicted predicted attitude of the object Tg. The image processing device P1 generates a template display screen SC in which the position of the icon PP2 and the prediction template TP2 are updated, and transmits it to the display 13 to display it.

[0096] Icon PP3 is the position (angle) of the camera CM capable of capturing an image of the template (2D) of the 3D model MD of the detection template TP3, and indicates the imaging position of the object Tg detected from the captured image on which image processing has been performed. The image processing device P1 updates the position of the icon PP3 and the detection template TP3 displayed in the fifth display area AR5 based on the posture of the object Tg detected by the first processing. The image processing device P1 generates a template display screen SC in which the position of the icon PP3 and the detection template TP3 are updated, and transmits it to the display 13 for display.

[0097] The second display area AR2 includes a 3D model MD of at least one object stored in the 3D model database DB. The second display area AR2 shown in Fig. 7 includes a 3D model MD (3D) of an object "A12", a 3D model MD (3D) of an object "A13", a 3D model MD (3D) of an object "A14", and a 3D model MD (3D) of an object "A15".

[0098] The second display area AR2 can receive a user operation to select one of the objects via the operation device 14. The image processing device P1 receives a user operation to select one of the selection areas SL1, SL2, SL3, and SL4 corresponding to each object, and displays a 3D model corresponding to any one of the selection areas SL1 to SL4 specified by the user operation in the first display area AR1.

[0099] Furthermore, the image processing device P1 in the second modification of the first embodiment accepts a user operation on the icon PP1 displayed in the first display area AR1, and displays a 3D model template (2D) corresponding to the position of the icon PP1 in the third display area AR3 based on the position of the icon PP1 moved by the user operation. When the registration button BT is selected (pressed) by the user operation, the image processing device P1 updates (registers) the template (2D) displayed in the third display area AR3 as a template to be used for feature matching in the second process.

[0100] The third display area AR3 includes a template TP1 (2D) when the 3D model MD of the object Tg is imaged from the position (angle) of the icon PP1.

[0101] The fourth display area AR4 includes the position (angle) of the icon PP2, that is, the prediction template TP2 (2D) when the 3D model MD of the object Tg is imaged from the predicted posture.

[0102] The fifth display area AR5 includes the position (angle) of the icon PP3, that is, the detection template TP3 (2D) when the 3D model MD of the object Tg is imaged from the posture in which the object Tg is detected.

[0103] The registration button BT is a button capable of accepting the generation of a template (2D) corresponding to the icon PP1 moved based on a user operation.

[0104] (First Modification of First Embodiment) The picking system 100 according to the first embodiment has been described as an example of performing template prediction by 3D matching using a 3D model. The picking system 100 according to the first modification of the first embodiment will be described as an example of performing template prediction based on the position of the target object Tg detected in the first process, information on the movement of the target object Tg obtained in multiple second processes performed while the first process is being performed, and captured images used in multiple second processes.

[0105] Note that an internal configuration example of the picking system 100 according to the first modification of the first embodiment has almost the same configuration as the internal configuration example of the picking system 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0106] Next, an overall operation procedure of the picking system 100 will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining an example of an overall operation procedure of the picking system 100 according to the first modification of the first embodiment.

[0107] The overall operation procedure example of the picking system 100 shown in Fig. 8 is an example, and is not limited thereto. Fig. 8 shows an example in which the first process is executed once and the second process is executed N times in picking one target Tg in order to easily understand the relationship between the first process and the second process, but the number of times the first process and the second process are executed is not limited thereto. It goes without saying that the picking system 100 may execute the first process multiple times in picking one target Tg.

[0108] The actuator AC picks up the object Tg conveyed on the belt conveyor by the camera CM while capturing images of the object Tg at a predetermined frame rate (for example, 1000 fps). The actuator AC shown in Fig. 8 shows a part of the picking process of the object Tg (time t11 to time t1N), and is executed repeatedly until the object Tg is picked up, for example.

[0109] The image processing device P1 acquires a captured image of the object Tg captured by the camera CM at a predetermined frame rate, and executes a second process (step St100) on the acquired captured image.

[0110] In parallel with the second process, the image processing device P1 executes the first process (step St200A) based on the captured image captured by the camera CM, information related to the movement of the object Tg, such as the motion information, obtained by the second process executed multiple times during the first process, and the captured image used in the second process multiple times. The image processing device P1 feeds back to the second process the prediction template TP2 (see FIG. 7), which is a template candidate for the object Tg obtained by the first process.

[0111] In the example shown in FIG. 8, the camera CM captures an object Tg at time t11 and transmits the captured image Img11 to the image processing device P1. The image processing device P1 acquires the first captured image Img11 (image data) transmitted from the camera CM and executes the first process and the second process using the first captured image Img11. The image processing device P1 outputs the motion information (information related to movement) of the object Tg acquired by the second process and the first captured image Img11 used in the second process to the first processing unit 110, and transmits information on the predicted position (x1, y1, z1) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired three-dimensional predicted position (x1, y1, z1) of the object Tg.

[0112] At time t12, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img12 to the image processing device P1. The image processing device P1 acquires the second captured image Img12 (image data) transmitted from the camera CM and executes a second process using the second captured image Img12. The image processing device P1 outputs the motion information (information related to movement) of the object Tg acquired by the second process and the second captured image Img12 used in the second process to the first processing unit 110, and transmits information on the predicted position (x2, y2, z2) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position (x2, y2, z2) of the object Tg.

[0113] After that, the picking system 100 repeatedly executes the same process from time t13 to time t1(N-1).

[0114] At time t1N, the camera CM captures an image of the object Tg. The camera CM transmits the captured image Img1N to the image processing device P1. The image processing device P1 acquires the N-th captured image Img1N (image data) transmitted from the camera CM and executes the second process using the N-th captured image Img1N. The image processing device P1 outputs the motion information (information related to movement) of the object Tg acquired by the second process and the N-th captured image Img1N used in the second process to the first processing device 110, and transmits information on the predicted position (xN, yN, zN) of the object Tg to the actuator AC. The actuator AC moves the end effector EF toward the acquired predicted position (xN, yN, zN) of the object Tg to pick up the object Tg. The image processing device P1 feeds back the template Img31 of the object Tg selected by the user operation to the second processing device 120.

[0115] As described above, the image processing device P1 in the first modification of the first embodiment can perform template prediction without using the 3D model MD of the object Tg.

[0116] (Modification 2 of the First Embodiment) The picking system 100 according to the first embodiment has shown an example in which a template is predicted using the matching tendency obtained by the second process and information related to the movement of the target object Tg. The picking system 100 according to the second modification of the first embodiment will be described below with reference to an example in which a template is generated using a template TP1 obtained based on a user operation.

[0117] Note that an example of an internal configuration of the picking system 100 according to the second modification of the first embodiment has almost the same configuration as the example of the internal configuration of the picking system 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0118] Next, the first process of the image processing device P1 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the first process procedure (step St200B) of the image processing device P1 in the second modification of the first embodiment.

[0119] 9 is similar to the operation procedure example of the first processing shown in FIG. 6 in steps St31 to St32 and steps St34 to St38, and therefore will not be described.

[0120] The first processing unit 110 determines whether or not a template TP1 of the target object Tg based on a user operation of the icon PP1 and pressing the registration button BT has been registered (St30A).

[0121] If the first processing unit 110 determines in the processing of step St30A that the template TP1 of the object Tg is registered (St30A, YES), it feeds back the registered template TP1 to the second processing unit 120 instead of the prediction template TP2 (St30B).

[0122] On the other hand, if the first processing unit 110 determines in the process of step St30A that the template TP1 of the object Tg has not been registered (St30A, NO), it acquires the captured image output from the second processing unit 120 (St31).

[0123] As described above, the image processing device P1 in the second modification of the first embodiment performs feature matching using the template TP1 specified by the user instead of the prediction template TP2, thereby improving the feature matching accuracy by the second processing unit 120 and also improving the tracking accuracy of tracking the target object Tg.

[0124] As described above, the image processing device P1 according to the first embodiment and the first modification of the first embodiment is movable and capable of communicating with the camera CM capable of capturing an image of the object Tg, and executes a first process (steps St200, 200A) of capturing an image of the object Tg, detecting the position or posture of the object Tg from the captured image, and generating a prediction template TP2 (an example of a template) of the object Tg, and executes a second process multiple times during the first process of performing feature matching (an example of template matching) based on the captured image and the prediction template TP2 of the object Tg and acquiring information regarding the movement of the object Tg. The first process predicts the posture of the object Tg based on the detected position or posture of the object Tg and the information regarding the movement of the object Tg acquired multiple times in the second process, and generates a prediction template TP2 of the object Tg corresponding to the predicted posture of the object Tg.

[0125] As a result, the image processing device P1 according to embodiment 1 and variant example 1 of embodiment 1 generates a prediction template TP2 by the first process, and tracks the changes in posture of the object Tg that change during the first process based on the feature matching result and the position information of the object Tg output from the second processing unit 120, and can generate a template (prediction template TP2) of the object Tg that is closest to the posture of the actual object Tg.

[0126] Moreover, the image processing device P1 according to the first embodiment and the first modification of the first embodiment extracts a feature amount of the object Tg from the captured image, and executes feature matching based on the extracted feature amount of the object Tg and the feature amount of the object Tg appearing in the prediction template TP2. This allows the image processing device P1 according to the first embodiment and the first modification of the first embodiment to acquire the position of the object Tg appearing in the captured image by position fitting based on feature matching.

[0127] Furthermore, the first process in the image processing device P1 according to the first embodiment executes 3D matching based on the object Tg detected from the captured image and the 3D model MD recorded in the 3D model database DB (an example of a database) to identify the posture of the object Tg, predicts the posture of the object Tg based on information on the movement of the object Tg acquired by the second process executed multiple times, and generates a prediction template TP2 of the object Tg. This allows the image processing device P1 according to the first embodiment to acquire the posture of the object Tg appearing in the captured image.

[0128] Furthermore, the image processing device P1 according to the first embodiment acquires a first imaging position of the camera CM with respect to the 3D model MD based on the predicted orientation of the object Tg, generates a prediction template TP2 corresponding to the predicted orientation based on the 3D model MD and the predicted orientation of the object Tg, associates the 3D model MD, the first imaging position with respect to the 3D model MD, and the prediction template TP2, and outputs the association to the display 13. This allows the image processing device P1 according to the first embodiment to visualize to the user that the prediction template TP2 is a template (2D image) when the 3D model MD of the object Tg is imaged from the first imaging position. Based on the prediction template TP2 and the first imaging position, the user can visually confirm whether the 3D model MD of the object Tg recognized by the image processing device P1 is the correct 3D model MD.

[0129] Moreover, the image processing device P1 according to the first embodiment detects the object Tg from the captured image, identifies the detected attitude of the detected object Tg, acquires the second imaging position of the camera CM relative to the 3D model MD based on the detected attitude, generates a detection template TP3 corresponding to the detected attitude based on the 3D model MD and the detected attitude, associates the 3D model MD, the second imaging position relative to the 3D model MD, and the detection template TP3, and outputs them to the display 13. As a result, the image processing device P1 according to the first embodiment can visualize to the user that the detection template TP3 is a template (2D image) when the 3D model MD of the object Tg is imaged from the second imaging position. Based on the detection template TP3 and the second imaging position, the user can visually confirm whether the 3D model MD of the object Tg recognized by the image processing device P1 is the correct 3D model MD.

[0130] Moreover, the image processing device P1 according to the first embodiment acquires designation information for designating the 3D model MD, and generates a prediction template TP2 of the object Tg based on the 3D model MD corresponding to the designation information. As a result, the image processing device P1 according to the first embodiment generates a template with a clean background other than the object Tg, that is, with less noise, thereby enabling more accurate position identification in the feature matching process of the second process. Moreover, the image processing device P1 generates a prediction template TP2 in which the position or posture of the object Tg that changes during the first process is corrected, thereby enabling real-time tracking of changes in posture or position of the object Tg captured by the camera CM.

[0131] Moreover, the image processing device P1 according to the second modification of the first embodiment acquires designation information for designating the template TP1, extracts features of the object Tg from the captured image, and performs feature matching based on the extracted features of the object Tg and the features of the object Tg appearing in the template TP1 corresponding to the designation information. This allows the image processing device P1 according to the second modification of the first embodiment to acquire the position of the object Tg appearing in the captured image by position fitting based on feature matching.

[0132] Furthermore, in the image processing device P1 according to the first embodiment and the first and second modifications of the first embodiment, the second processing is executed at a frame rate different from that of the first processing for the captured image captured by the camera CM. As a result, the image processing device P1 according to the first embodiment and the first and second modifications of the first embodiment can track in real time the change in posture or position of the target object Tg captured by the camera CM by combining an advanced low-speed image processing (first processing) with a low-speed image processing technique (second processing).

[0133] Moreover, the image processing device P1 according to the first embodiment and the first modification of the first embodiment includes a communication unit 10 (an example of an acquisition unit) that acquires a captured image of the object Tg captured by a camera CM that is movable and capable of capturing an image of the object Tg, a first processing unit 110 that detects the position or posture of the object Tg from the captured image and generates a template of the object Tg, and a second processing unit 120 that performs template matching based on the captured image and the template of the object Tg multiple times while the first processing unit 110 is generating the template of the object Tg, and acquires information regarding the movement of the object Tg. The first processing unit 110 predicts the posture of the object Tg based on the detected position or posture of the object Tg and the information regarding the movement of the object Tg acquired multiple times by the second processing unit 120, and generates a template of the object Tg corresponding to the predicted posture of the object Tg.

[0134] As a result, the image processing device P1 according to embodiment 1 and variant example 1 of embodiment 1 generates a prediction template TP2 by the first process, and is able to track the change in posture of the object Tg that changes during the first process based on the feature matching result and the position information of the object Tg output from the second processing unit 120, and generate a template (prediction template TP2) of the object Tg that is closest to the posture of the actual object Tg.

[0135] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, corrections, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also belong to the technical scope of the present disclosure. In addition, the components in the various embodiments described above may be arbitrarily combined within the scope of the invention. [Industrial Applicability]

[0136] The present disclosure is useful as an image processing method and image processing device that registers a highly accurate template of an object that can be used for template matching even in a situation where the orientation of the object from an imaging device changes as the imaging device moves. [Explanation of symbols]

[0137] 10. Communications Department 11 Processors 12. Memory 13. Display 14 Operating Devices 110 First Processing Section 120 Second Processing Section AC Actuator Commercial Camera DB 3D model database MD 3D models P1 Image Processing Device Tg Object

Claims

1. An image processing method performed by an image processing device that is movable and capable of communicating with a camera that can capture an image of an object, comprising: A first process is executed to acquire a captured image of the object, detect a position or a posture of the object from the captured image, and generate a template of the object; executing a second process a plurality of times to execute template matching based on the captured image and a template of the object while executing the first process, and acquiring information regarding a movement of the object; The first process predicts a posture of the object based on the detected position or posture of the object and information on the movement of the object acquired multiple times in the second process, and generates a template of the object corresponding to the predicted posture of the object. Image processing methods.

2. extracting a feature amount of the object from the captured image, and performing template matching based on the extracted feature amount of the object and the feature amount of the object captured in the template; The image processing method according to claim 1 .

3. The first process performs 3D matching between the object detected from the captured image and a 3D model recorded in a database to identify a posture of the object, predicts the posture of the object based on information on the movement of the object acquired in the second process executed multiple times, and generates a template of the object. The image processing method according to claim 1 .

4. obtaining a first imaging position of the camera relative to the 3D model based on the predicted pose of the object; generating a predictive template based on the 3D model and the predicted pose of the object, the predictive template corresponding to the predicted pose; outputting the 3D model, the first imaging position with respect to the 3D model, and the prediction template to a display in association with each other; The image processing method according to claim 3.

5. Detecting the object from the captured image and identifying a detected attitude of the detected object; obtaining a second imaging position of the camera relative to the 3D model based on the detected orientation; generating a detection template corresponding to the detected pose based on the 3D model and the detected pose; outputting the 3D model, the second imaging position with respect to the 3D model, and the detection template to a display in association with each other; The image processing method according to claim 3.

6. Acquire designation information for designating the 3D model; generating a template of the object based on a 3D model corresponding to the specified information; The image processing method according to claim 3.

7. acquiring designation information designating the template; Extracting a feature amount of the object from the captured image; performing template matching based on the extracted feature amount of the object and the feature amount of the object depicted in a template corresponding to the designation information; The image processing method according to claim 1 .

8. The second processing is executed at a frame rate different from that of the first processing for the captured image captured by the camera. The image processing method according to claim 1 .

9. an acquisition unit that acquires an image of an object captured by a movable camera that can capture an image of the object; a first processing unit that detects a position or an orientation of the object from the captured image and generates a template of the object; a second processing unit that performs template matching based on the captured image and the template of the object a plurality of times while the template of the object is being generated by the first processing unit, and acquires information regarding a movement of the object; The first processing unit predicts a posture of the object based on the detected position or posture of the object and information on the movement of the object acquired multiple times by the second processing unit, and generates a template of the object corresponding to the predicted posture of the object. Image processing device.

Citation Information

Patent Citations

  • Template preparation device and template preparation method

    JP2016207147A