Recognition device, robot control system, recognition method, and program

Abstract

To provide a technology which can reduce the number of processing, when estimating a position and attitude of an object, by using a measurement image obtained by measuring the object.SOLUTION: A recognition device recognizes a position and attitude of a predetermined object in a measurement image, by comparing the measurement image showing a result obtained by measuring a measurement range including the predetermined object with a three-dimensional model of the predetermined object. The recognition device comprises: estimation means which estimates the position and attitude of the predetermined object in the present, based on the position and attitude of the predetermined object which the recognition device has recognized in the past; and recognition means which assumes the position and attitude which the estimation means has estimated as an initial value of the position and attitude of the three-dimensional model of the predetermined object, and recognizes the position and attitude of the predetermined object in the measurement image, depending on the result obtained by matching the three-dimensional model to the predetermined object in the measurement image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a recognition device, a robot control system, a recognition method, and a program. [Background technology]

[0002] Conventionally, the position and orientation of an object is recognized (estimated) by comparing a template that indicates the characteristics of the object in each orientation with an image (measurement image) captured by an imaging device of the object.

[0003] Patent Document 1 discloses a system that changes the size of a template according to the distance obtained from a distance sensor and performs matching. This makes it possible to reduce the number of templates held by the system. In addition, by selecting a template according to the distance, the number of matching operations can be reduced. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Pat. No. 9,659,217 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if the number of templates is reduced as in Patent Document 1, a template must be prepared for each object orientation and each template must be matched with an image, which still requires many matches, and therefore requires a lot of processing to estimate the position and orientation of an object.

[0006] Therefore, an object of the present invention is to provide a technique capable of reducing the number of processes when estimating the position and orientation of an object using a measurement image obtained by measuring the object. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention employs the following configuration.

[0008] That is, a recognition device according to one aspect of the present invention is a recognition device that recognizes the position and orientation of a specified object in a measurement image by comparing a measurement image representing the results of measuring a measurement range including the specified object with a three-dimensional model of the specified object, and is characterized in having: an estimation means that estimates the current position and orientation of the specified object based on the position and orientation of the specified object recognized by the recognition device in the past; and a recognition means that recognizes the position and orientation of the specified object in the measurement image based on the result of aligning the three-dimensional model to the specified object in the measurement image, using the position and orientation estimated by the estimation means as initial values ​​for the position and orientation of the three-dimensional model of the specified object.

[0009] According to this configuration, the position and orientation of an object can be estimated quickly without using a template. Furthermore, since the position and orientation of an object can be roughly grasped before comparing the 3D model with a measurement image, the range of the position and orientation of the object of the 3D model for comparison can be limited. Therefore, the position and orientation of an object can be recognized more quickly and efficiently.

[0010] The recognition device further includes a setting unit for setting the measurement range, and the estimation unit determines whether the predetermined object exists based on the position or position and orientation estimated by the estimation unit. The measurement range may be set to the estimated range by the setting means, and the measurement range may be set to the estimated range by the setting means. This makes it possible to limit the measurement range, thereby making it possible to acquire a measurement image at a higher speed.

[0011] In the above recognition device, the estimation means may estimate the current position and orientation of the predetermined object based on a change in the position and orientation of the predetermined object between at least two past points in time. This allows the position and orientation of the predetermined object to be estimated in a simple manner (i.e., with fewer steps). This makes it possible to recognize the position and orientation of an object more quickly and efficiently.

[0012] In the above recognition device, the measurement image may represent a result of measurement by an image sensor, the position and orientation of the image sensor may be controlled by a robot, and the estimation means may estimate the current position and orientation of the predetermined object based on the past position and orientation of the predetermined object and the current position and orientation of the robot. This makes it possible to estimate the current position and orientation of the predetermined object even when only the position and orientation of the predetermined object at one past point in time can be used for estimation.

[0013] In the above recognition device, the measurement image may be a distance image in which each pixel represents a distance to a subject.

[0014] In the above recognition device, when there is no recognition result of the past position and orientation of the predetermined object by the recognition means, the estimation means may not estimate the current position and orientation of the predetermined object, and the recognition means may recognize the position and orientation of the predetermined object according to a comparison result between each of a plurality of templates indicating feature amounts of the predetermined object and the measurement image. In this way, even when there is no recognition result of the past position and orientation of the predetermined object, it is possible to estimate the current position and orientation of the predetermined object.

[0015] The present invention may also be a robot control system including the above-mentioned recognition device, an image sensor that measures the measurement range, and a robot that controls the position and orientation of the predetermined object based on the results of recognition by the recognition device. With this, the position and orientation of the predetermined object can be recognized at high speed, and therefore the robot can be efficiently controlled.

[0016] The present invention may be understood as an apparatus having at least a part of the above means, or as an electronic device, a control system, an information processing system, an information processing device, a recognition device, a recognition system, or a robot. The present invention may also be understood as a control method, a collation method, a recognition method, or a robot control method including at least a part of the above processing. The present invention may also be understood as a program for implementing such a method, or a recording medium (storage medium) on which the program is non-temporarily recorded. Note that the above means and processing can be combined with each other as much as possible to constitute the present invention. Effect of the Invention

[0017] According to the present invention, when the position and orientation of an object are estimated using a measurement image obtained by measuring the object, the number of processes can be reduced. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram illustrating a recognition system according to a first embodiment. [Diagram 2] FIG. 2 is a configuration diagram of the template generating device and the recognition device according to the first embodiment. [Diagram 3] FIG. 3 is a flowchart of the object position and orientation recognition process according to the first embodiment. [Figure 4] FIG. 4 is a flowchart of the recognition process using past information according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart of the recognition process using past information according to the second embodiment. [Figure 6] FIG. 6 is a diagram for explaining the coordinate system of the recognition system. [Figure 7] FIG. 7 is a diagram illustrating another configuration of the recognition system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

[0020] <Application Examples> In the following, a recognition system 1 that compares a template indicating the feature amount of an object based on a 3D model (3D data) of the object with an image (measurement image) showing the result of measuring the object by an imaging sensor estimates the current position and orientation of the object according to the past position and orientation of the object. Then, the recognition system 1 recognizes the position and orientation of the object by comparing the 3D model (3D data) of the object with the estimated position and orientation as the initial value and the measurement image. Note that the position and orientation means the position and orientation in the following embodiment, but it may be the position or the orientation as long as there is no technical contradiction.

[0021] This allows the position and orientation of an object to be estimated quickly without using a template. Furthermore, since the position and orientation of an object can be roughly grasped before comparing the 3D model with a measurement image, the range of the position and orientation of the object in the 3D model for comparison can be limited. Therefore, the position and orientation of an object can be recognized faster and more efficiently.

[0022] <Embodiment 1> [Configuration of the recognition system] The configuration of a recognition system 1 according to a first embodiment will be described with reference to Fig. 1. The recognition system 1 recognizes the position and orientation of an object 2 (a specified object) by comparing a three-dimensional model of the object 2 with a measurement image showing the result of measuring the object 2. The recognition system 1 includes an image sensor 10, a template generating device 20, a recognition device 30, a storage device 40, a robot control device 50, and a robot 60. The recognition system 1 is also a robot control system that controls the robot 60 that connects a grasped object 3 to the object 2.

[0023] The imaging sensor 10 acquires a measurement image by measuring a measurement range including the current object 2. In this embodiment, the imaging sensor 10 is a three-dimensional sensor that acquires an image of the subject captured through a left lens and an image of the subject captured through a right lens, and acquires a distance image as a measurement image by comparing the difference between the two images (left and right images). In the distance image, each pixel indicates the distance from the imaging sensor 10 to the subject. The imaging sensor 10 may acquire the distance image by any method such as triangulation measurement or a ToF (Time of Flight) method. The measurement image may be a temperature image in which each pixel indicates the temperature of the subject, or may be a normal optical image (an image expressing the color and brightness of the subject).

[0024] The template generating device 20 generates a template indicating feature quantities of the object 2 based on a 3D model of the object 2 measured in advance or a 3D model of the object 2 used when designing the object 2. For example, the 3D model (3D data) may be data expressing the object 2 using point cloud data, or data defining the shape and structure of the object 2 such as CAD data.

[0025] The recognition device 30 uses the measurement images acquired by the image sensor 10 to determine the current position and posture of the object 2. Based on the result of the estimation, the recognition device 30 recognizes the current position and orientation of the object 2.

[0026] The storage device 40 stores (records) the 3D model of the object 2, the template generated by the template generation device 20, the matching result obtained by the recognition device 30, and / or information on the position and orientation of the object 2 recognized by the recognition device 30. The storage device 40 may be a server having a hard disk (HDD) or memory (RAM; Random Access Memory), or the like. The storage device 40 may also be a storage medium that is removable from the template generation device 20 and the recognition device 30. The storage device 40 stores a plurality of templates that represent the features of the object 2 in different orientations (the object 2 viewed from different viewpoints).

[0027] The robot control device 50 controls the posture (position and posture) of the robot 60 based on the recognition result of the position and posture of the object 2. The robot control device 50 controls the posture of the robot 60 so that the object 3 grasped by the grasping unit 61 of the robot 60 is connected to the object 2.

[0028] The posture of the robot 60 is controlled by the robot control device 50. The robot 60 has a gripping part 61 that grips the object 3, and a base 62 to which an end of the robot 60 is fixed. Therefore, the robot 60 can control the position and posture of the object 3. In addition, the image sensor 10 is fixed (connected) to a part of the gripping part 61.

[0029] (Internal configuration of the template generation device) 2, the internal configuration of the template generating device 20 will be described. The template generating device 20 has a control unit 201, an information acquiring unit 202, a projection image generating unit 203, a feature amount calculating unit 204, a template generating unit 205, and an information output unit 206.

[0030] The control unit 201 controls each functional unit of the template generating device 20 according to a program non-temporarily stored in a storage medium.

[0031] The information acquisition unit 202 acquires a three-dimensional model (three-dimensional data; three-dimensional image data) of the object 2. The information acquisition unit 202 may acquire the three-dimensional model of the object 2 from the storage device 40 or from another external device. The information acquisition unit 202 also acquires imaging parameters of the image sensor 10 (camera focal length, center coordinates of the image, and lens distortion correction coefficient).

[0032] The projection image generating unit 203 generates a projection image by converting a three-dimensional model into a two-dimensional image. Specifically, the projection image generating unit 203 generates a projection image that represents the object 2 in each posture by a two-dimensional image. At this time, since the measurement image changes depending on the imaging parameters of the image sensor 10, the projection image generating unit 203 generates a projection image corrected by the imaging parameters.

[0033] The feature amount calculation unit 204 calculates the feature amount of each pixel (each region) in the projection image based on the three-dimensional model or the projection image. Here, the feature amount can be an edge feature amount (edge ​​direction histogram) or a normal feature amount (normal direction histogram). However, the feature amount is not limited to this, and can be distance information, temperature information, or color information.

[0034] The template generating unit 205 generates a template which is a two-dimensional image in which each pixel indicates the feature amount calculated by the feature amount calculating unit 204. The template generating unit 205 also adds, to the template, information on the posture of the object 2 corresponding to the projected image on which the template is based.

[0035] Information output unit 206 outputs the template generated by template generation unit 205 to storage device 40.

[0036] In this embodiment, the projection image generating unit 203 generates a plurality of projection images from a three-dimensional model. At this time, the plurality of projection images are images in which the object 2 in different postures is represented by two-dimensional images. Thereafter, the feature amount calculating unit 204 calculates a feature amount for each of the plurality of projection images. Then, the template generating unit 205 generates a template for each of the plurality of projection images, and the information output unit 206 outputs the plurality of templates to the storage device 40.

[0037] (Internal structure of the recognition device) 2, the internal configuration of the recognition device 30 will be described. The recognition device 30 has a control unit 301, an image acquisition unit 302, an information acquisition unit 303, a feature amount calculation unit 304, a range setting unit 305, an estimation unit 306, a matching unit 307, a recognition unit 308, and a result output unit 309.

[0038] The control unit 301 controls each functional unit of the recognition device 30 in accordance with a program non-temporarily stored in a storage medium.

[0039] The image acquiring section 302 acquires a measurement image from the imaging sensor 10. Note that the image acquiring section 302 does not need to acquire a measurement image from the imaging sensor 10, and may acquire a measurement image stored in the storage device 40, for example.

[0040] The information acquisition unit 303 acquires a plurality of templates from the storage device 40 .

[0041] The feature amount calculation unit 304 calculates the feature amount of each pixel (each region) of the measurement image.

[0042] The range setting unit 305 sets the range in which the object 2 may exist (the estimated range estimated by the estimation unit 306) as the range of the object to be measured by the image sensor 10 (measurement range).

[0043] The estimation unit 306 estimates the current position and orientation of the object 2 based on the past position and orientation of the object 2. The estimation unit 306 does not execute template matching, which requires time and effort. Therefore, the estimation unit 306 can estimate the position and orientation of the object 2 faster than estimating the position and orientation of the object 2 by matching with a template. The estimation unit 306 also estimates a range (estimation range) in which the object 2 may exist, from the estimated position or position and orientation of the object 2. The estimation range is, for example, a range of any shape, such as a rectangular parallelepiped, cylindrical, or spherical shape of a predetermined size, centered on the estimated position of the object 2.

[0044] The matching unit 307 matches the template acquired by the information acquisition unit 303 with the measurement image. If the degree of match is greater than a predetermined value, the matching unit 307 determines that the matching is successful. On the other hand, if the degree of match is equal to or less than the predetermined value, the matching unit 307 determines that the matching is unsuccessful. Furthermore, the matching unit 307 estimates (recognizes) a rough position and orientation of the object 2 from the template when it is determined that the matching is successful. Specifically, the matching unit 307 can estimate the position and orientation of the object 2 based on the information on the orientation of the object 2 added to the template and the information on the orientation of the robot 60 at the time of measurement.

[0045] The recognition unit 308 uses the position and orientation of the object 2 estimated by the estimation unit 306 or the matching unit 307 as the initial value of the position and orientation of the object 2 in the three-dimensional model, and matches the object 2 in the three-dimensional model to the object 2 appearing in the measurement image (measurement data). This matching can be realized, for example, by comparing each point of the object 2 in the three-dimensional model with the object 2 in the measurement image using an ICP (Iterative Closest Point) algorithm. Based on the result of fitting to the measurement image, the detailed position and orientation of the object 2 in the measurement image is recognized.

[0046] The result output unit 309 outputs the matching result and the recognition result of the position and orientation of the object 2 to the storage device 40 or an external device. If the result output unit 309 outputs the recognition result to a robot control device 50 for driving the object 2, the robot control device 50 can perform control to set the robot 60 (grasped object 3) to a predetermined position and orientation according to the position and orientation of the object 2.

[0047] Furthermore, the template generating device 20 and the recognition device 30 can be configured by a computer including, for example, a CPU (processor), a memory, a storage, and the like. In this case, the configuration shown in FIG. 2 is realized by loading a program stored in the storage into the memory and executing the program by the CPU. Such a computer may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smartphone, or may be an embedded computer such as an on-board computer. Alternatively, all or a part of the configuration shown in FIG. 2 may be configured by ASIC, FPGA, or the like. Alternatively, all or a part of the configuration shown in FIG. 2 may be realized by cloud computing or distributed computing.

[0048] [Object position and orientation recognition processing] The recognition process in which the recognition device 30 recognizes the position and orientation of the object 2 will be described with reference to the flowchart in Fig. 3. Each process in the flowchart in Fig. 3 can be realized by the control unit 301 executing a program.

[0049] In step S1001, the control unit 301 controls the image acquisition unit 302 to acquire left and right images (an image of the subject captured through the left lens and an image of the subject captured through the right lens) from the imaging sensor 10. The control unit 301 also controls the information acquisition unit 303 to acquire a plurality of templates and a three-dimensional model of the object 2 from the storage device 40. Here, the plurality of templates are templates (multiple viewpoint templates) corresponding to different postures of the object 2. The control unit 301 may acquire imaging parameters from the imaging sensor 10.

[0050] In step S1002, the control unit 301 determines whether or not the past (past frame) recognition result of the position and orientation of the object 2 is stored in the storage device 40. Note that the past (past frame) recognition result of the position and orientation of the object 2 may be stored in a storage unit (not shown) of the recognition device 30, and the control unit 301 may determine whether or not the past recognition result of the position and orientation of the object 2 is stored in the storage unit. If it is determined that the past recognition result of the position and orientation of the object 2 is stored, the process proceeds to step S1006, and if not, the process proceeds to step S1003.

[0051] In step S1003, the control unit 301 controls the image sensor 10 to measure the distance to the entire overlapping range of the left and right images, and acquires a measurement image (performs three-dimensional measurement). Specifically, the image sensor 10 identifies corresponding pixels between the two left and right images, and calculates the difference in position of the corresponding pixels. Then, the image sensor 10 measures the distance to the subject using triangulation technology based on the difference in position of the corresponding pixels and the difference in position of the left and right lenses. In this way, the image sensor 10 can acquire a measurement image that is, for example, a distance image having point cloud data.

[0052] In step S1004, the control unit 301 controls the feature amount calculation unit 304 to acquire the feature amount from the measurement image.

[0053] In step S1005, the control unit 301 controls the matching unit 307 to estimate (recognize) a rough position and orientation of the object 2 using the multiple templates acquired in step S1001. Specifically, first, the matching unit 307 performs matching by comparing feature amounts indicated by the multiple templates with feature amounts indicated by the measurement image. For example, the matching unit 307 acquires the reciprocal 1 / Sum(D) of the total sum Sum(D) of the difference D of feature amounts between each pixel in a certain range of the measurement image and the template as the degree of match. Note that the method is not limited to this, and the degree of match may be acquired by any matching method. Then, the matching unit 307 estimates the orientation of the object 2 based on the orientation corresponding to the template with the highest degree of match and the orientation of the robot 60 (the position and orientation of the image sensor 10). The matching unit 307 also estimates the position of the object 2 based on the position in the measurement image that most matches the template and the orientation of the robot 60.

[0054] In step S1006, the control unit 301 executes a recognition process using information (past information) of the past position and orientation of the object 2. Details of the recognition process using the past information will be described later with reference to the flowchart in FIG.

[0055] In step S1007, the control unit 301 determines whether or not the position and orientation of the object 2 have been recognized by the recognition process in step S1006. If it is determined that the position and orientation of the object 2 have been recognized, the process proceeds to step S1009; otherwise, the process proceeds to step S1003.

[0056] In step S1008, the control unit 301 controls the recognition unit 308 to compare the 3D model of the object 2 with the measurement image and recognize the position and orientation of the object 2 in the measurement image in detail. Specifically, the recognition unit 308 recognizes the position and orientation of the object 2 in the measurement image by fitting the 3D model of the object 2 to the measurement image using the position and orientation estimated by the matching unit 307 as initial values. The recognition unit 308 can fit the 3D model of the object 2 to the measurement image by comparing (matching) each point of the object 2 shown in the 3D model and the measurement image using, for example, an ICP algorithm.

[0057] In step S1009, the control unit 301 controls the result output unit 309 to output the information on the position and orientation of the object 2 to the storage device 40. As a result, the information on the position and orientation of the object 2 is stored in the storage device 40.

[0058] (Recognition process using past information; step S1006) The recognition process using past information performed in step S1006 will be described with reference to the flowchart of Fig. 4. Each process in the flowchart of Fig. 4 can be realized by the control unit 301 executing a program.

[0059] In step S2001, the control unit 301 controls the estimation unit 306 to calculate the amount of change (amount of movement and amount of rotation) in the position and orientation of the object 2 between two past frames of the measurement image based on the past recognition result of the position and orientation of the object 2. Specifically, the estimation unit 306 may calculate the difference in the position and orientation of the object 2 between the two frames as the amount of change. Note that the estimation unit 306 may calculate the amount of change in the position and orientation of the object 2 between frames based on the recognition results of the position and orientation of the object at three or more points in time.

[0060] In step S2002, the control unit 301 controls the estimation unit 306 to estimate the amount of change (estimated amount of change; amount of movement and amount of rotation) of the object 2 between the previous frame and the current frame based on the amount of change of the object 2 between the past two frames of the measurement image. Specifically, for example, when the two past frames are separated by a plurality of frames, the estimation unit 306 estimates the amount of change of the object 2 between the past two frames of the measurement image based on the frame distance between the two past frames. The estimated change amount is calculated (estimated) by dividing by the number of frames.

[0061] In step S2003, the control unit 301 controls the estimation unit 306 to estimate the position and orientation of the object 2 in the current frame based on the position and orientation of the object 2 in the previous frame and the estimated amount of change of the object 2 between the previous frame and the current frame. The estimation unit 306 estimates that the object 2 has moved by the estimated amount of change from the position and orientation of the object 2 in the previous frame, and estimates the position and orientation of the object 2 in the current frame. In this way, in step S2003, the estimation unit 306 estimates the current position and orientation of the object 2 based on the past position and orientation of the object 2, instead of estimating the position and orientation of the object 2 by matching with a template in step S1005.

[0062] In step S2004, the control unit 301 controls the estimation unit 306 to estimate a range (estimated range) in which the object 2 may exist in the current frame based on the estimated current position and orientation of the object. For example, the estimation unit 306 determines a range of a predetermined size (e.g., 1.5 or 2 times the size of the object 2) centered on the position of the object 2 estimated in step 2003 as an estimated region. Then, the control unit 301 controls the range setting unit 305 to set the estimated region to the measurement range, which is the range measured by the image sensor 10.

[0063] In step S2005, the control unit 301 controls the image sensor 10 to measure only the measurement range (estimated range) set in step S2004, and acquires a measurement image. This allows only the measurement range in which the object 2 may exist to be measured, thereby making the measurement by the image sensor 10 more efficient.

[0064] Specifically, the image sensor 10 measures only the measurement range where the object 2 may exist by calculating the difference between the positions of corresponding pixels only in the range corresponding to the measurement range of the left and right images. Therefore, the number of pixels to be subjected to the calculation process of the difference between the positions of pixels in the left and right images can be reduced. In addition, since the distance to be measured from the image sensor 10 is limited, the range for searching the corresponding positions of the pixels of one image to the pixels of the other image is narrowed. In other words, the number of processes related to the calculation of the corresponding pixels between the left and right images is also reduced. In this way, the image sensor 10 can perform measurement only in a predetermined range including the position where the object 2 may exist. In other words, the image sensor 10 does not need to measure the position far from the position where the object 2 exists.

[0065] In step S2006, the control unit 301 controls the recognition unit 308 to compare the three-dimensional model of the object 2 with the measurement image, and recognize the position and orientation of the object 2 in detail. Estimation section 306 The recognition unit 308 recognizes the position and orientation of the object 2 in the measurement image by fitting the 3D model of the object 2 to the measurement image using the position and orientation estimated by the recognition unit 308 as initial values. The recognition unit 308 can fit the 3D model of the object 2 to the measurement image by comparing (matching) the 3D model of the object 2 with each point of the object 2 shown in the measurement image using an ICP algorithm.

[0066] 4 ends, and it is determined that the position and orientation of the object 2 could not be recognized, and the process proceeds to step S1003 in step S1007. The predetermined case may be, for example, a case where only the position and orientation of the object 2 in one past frame is stored, and the amount of change in the object 2 between the past frames cannot be calculated in step S2001. Another predetermined case may be a case where the 3D model of the object 2 cannot be fitted to the measurement image in step S2006 due to the object 2 not being present within the measurement range (estimation range).

[0067] According to this embodiment, the rough position of the object 2 is calculated based on the past position and orientation of the object 2. Since the orientation can be estimated, the process of matching the template with the measurement image can be eliminated, and therefore the recognition process can be made more efficient when recognizing the position and orientation of the object 2.

[0068] <Embodiment 2> In the second embodiment, a recognition system 1 that estimates the current position and orientation of an object 2 based on the position and orientation of a robot 60 will be described. The configuration of the recognition system 1 according to the second embodiment is similar to that of the recognition system 1 according to the first embodiment, and therefore a description thereof will be omitted. In the second embodiment, only the process of step S1006 shown in Fig. 2 (recognition process using past information) is different from the first embodiment, and therefore the recognition process using past information will be described with reference to the flowchart shown in Fig. 5.

[0069] Here, the coordinate systems in the recognition system 1 will be described. In the recognition system 1, as shown in FIG. 6, there are four coordinate systems: a base coordinate system, an object coordinate system, a tool coordinate system, and a camera coordinate system. The base coordinate system is a fixed coordinate system centered on the base 62 of the robot 60. The object coordinate system is a coordinate system centered on the object 2. The tool coordinate system is a coordinate system centered on the gripper 61 (or the gripped object 3) of the robot 60. The camera coordinate system is a coordinate system centered on the image sensor 10. Here, the "position and orientation of the object 2" according to the first embodiment may be the position and orientation of the object 2 in the camera coordinate system, or the position and orientation of the object 2 in the base coordinate system.

[0070] Fig. 5 is a flowchart showing the recognition process using past information according to embodiment 2. In Fig. 5, the processes in steps S2004 to S2006 are similar to those in steps S2004 to S2006 according to embodiment 1. Therefore, the description of steps S2004 to S2006 will be omitted.

[0071] In step S3001, the control unit 301 converts the position and orientation of the object 2 in the past frame of the camera coordinate system into the base coordinate system based on the position and orientation of the object 2 in the past frame of the camera coordinate system and the position and orientation of the robot 60 (gripping unit 61) in the past frame of the base coordinate system, using the estimation unit 306. Specifically, the control unit 301 converts the position (translation) T' of the object 2 in the base coordinate system of the past frame into the base coordinate system using the following equation 1: obj-base and rotation (posture) R' obj-base Get the.

number

[0072] In formula 1, position T' obj-cam and rotation R' obj-cam are the position (coordinates) and rotation (orientation) of object 2 in the previous frame of the camera coordinate system. obj-cam and rotation R' obj-cam is the past recognition result of the position and orientation of the object 2 by the recognition device 30.tool-base and rotation R' tool-base are the position and rotation of the gripper 61 (robot 60) in the past frame of the base coordinate system. cam-tool and rotation R cam-tool are the position and rotation of the image sensor 10 in the tool coordinate system. Note that the relative positional relationship between the image sensor 10 and the gripper 61 is constant, so position T cam-tool and rotation R cam-tool is a constant value regardless of whether it is a past frame or a current frame. cam-tool and rotation R cam-tool can be obtained by measuring in advance.

[0073] Here, the transformation vector from the object coordinate system to the base coordinate system is l, the transformation vector from the object coordinate system to the camera coordinate system is m, the transformation vector from the tool coordinate system to the camera coordinate system is n, and the transformation vector from the base coordinate system to the tool coordinate system is Let o. Then, l=m+n -1 +o -1 This can be expressed in terms of position and rotation as in Equation 2. This allows the calculation of Equation 1 to be calculated.

number

[0074] In step S3002, the control unit 301 uses the estimation unit 306 to estimate the position and orientation of the object 2 in the current frame in the camera coordinate system based on the position and orientation of the object 2 in the base coordinate system of the past frame calculated in step S3001 and the position and orientation of the robot 60 (base 62) in the current frame in the tool coordinate system. Specifically, the control unit 301 estimates the position T obj-cam and rotation (posture) R' obj-cam Get the.

number

[0075] position T base-tool and rotation R base-tool are the position (coordinates) and rotation (pose) of the base 62 (robot 60) in the current frame of the tool coordinate system. tool-cam and rotation R tool-cam are the coordinates of the grip part 61 in the camera coordinate system. Since the relative positional relationship between the image sensor 10 and the gripping part 61 is constant, the position T tool-cam and rotation R tool-cam is the past frame and the present This is a constant value regardless of the frame. Therefore, the position T tool-cam and rotation R to ol-cam can be obtained by measuring in advance.

[0076] In this way, steps S3001 and S3002 enable the estimation unit 306 to estimate the position and orientation of the object 2 in the current frame. Here, in the first embodiment, the estimation unit 306 needs to know the positions and orientations of the object 2 in multiple past frames (two time points) in order to estimate the position and orientation of the object in the current frame. On the other hand, according to the second embodiment, the estimation unit 306 can estimate the position and orientation of the object 2 in the current frame if it can know the position and orientation of the object 2 or the robot 60 in one past frame. In other words, the estimation unit 306 can estimate the position and orientation of the object 2 in the current frame by measurement at one time point.

[0077] Therefore, in the second embodiment, the recognition system 1 can execute the recognition process when recognizing the position and orientation of the object 2 more efficiently than in the first embodiment.

[0078] In the first and second embodiments, the recognition system 1 estimates the position and orientation of the object 2 based on the measurement images captured by the moving image sensor 10, as shown in Fig. 1, and recognizes the position and orientation of the object 2 based on the estimation result. However, as shown in Fig. 7, even if the position of the image sensor 10 is fixed and the position and orientation of the object 2 change over time, The recognition system 1 according to each embodiment can be applied. That is, even in such a case, the recognition system 1 can estimate the current position and orientation of the object 2 based on the past position and orientation of the object 2 by the method (estimation method: recognition method) described in the first or second embodiment.

[0079] Furthermore, the recognition system 1 (estimation unit 306) according to each embodiment may estimate the current position and orientation of the object 2 by further using settings for the object 2 (setting information on the movement speed, acceleration, and movement trajectory) in addition to the past posture of the object 2. Furthermore, the recognition system 1 (estimation unit 306) may estimate the current position and orientation of the object 2 based on settings for posture change of the robot 60 (encoder values ​​and acceleration of the joints, and setting information on the movement trajectory).

[0080] The interpretation of the claims is not limited solely to the matters described in the embodiments. The interpretation of the claims also includes the scope of the claims that is described in such a way that a person skilled in the art can recognize that the problem to be solved by the invention can be solved, taking into account the common general technical knowledge at the time of filing.

[0081] (Appendix 1) A recognition device (30) that recognizes a position and orientation of a predetermined object in a measurement image by comparing the measurement image, which represents a result of measuring a measurement range including the predetermined object, with a three-dimensional model of the predetermined object, comprising: an estimation means (306) for estimating a current position and orientation of the predetermined object based on a position and orientation of the predetermined object recognized by the recognition device (30) in the past; a recognition means (308) for recognizing a position and orientation of the predetermined object in the measurement image according to a result of fitting the three-dimensional model to the predetermined object in the measurement image, using the position and orientation estimated by the estimation means (306) as an initial value of the position and orientation of the three-dimensional model of the predetermined object; A recognition device (30) comprising:

[0082] (Appendix 2) 1. A method for recognizing a position and orientation of a predetermined object in a measurement image by comparing the measurement image, which represents a result of measuring a measurement range including the predetermined object, with a three-dimensional model of the predetermined object, comprising: an estimation step (S2001, S2002, S2003) of estimating a current position and orientation of the predetermined object based on a position and orientation of the predetermined object recognized in the past; a recognition step (S2006) of recognizing a position and orientation of the predetermined object in the measurement image according to a result of fitting the three-dimensional model to the predetermined object in the measurement image, using the position and orientation estimated in the estimation step as an initial value of the position and orientation of the three-dimensional model of the predetermined object; A recognition method comprising the steps of: [Explanation of symbols]

[0083] 1: Recognition system, 2: Object, 3: Grasped object, 10: imaging sensor, 20: template generating device, 30: recognition device, 40: storage device, 50: robot control device, 60: robot, 61: gripping portion, 62: base, 201: control unit, 202: information acquisition unit, 203: projection image generation unit, 204: feature amount calculation unit, 205: template generation unit, 206: information output unit, 301: control unit, 302: image acquisition unit, 303: information acquisition unit, 304: feature amount calculation unit, 305: range setting unit, 306: estimation unit, 307: Matching section, 308: Recognition section, 309: Result output section

Claims

1. 1. A recognition device that recognizes a position and orientation of a predetermined object in a measurement image by comparing the measurement image, which represents a result of measuring a measurement range including the predetermined object, with a three-dimensional model of the predetermined object, comprising: an estimation means for estimating a current position and orientation of the predetermined object based on past information when past information is present, the past information being a result of recognition of the position and orientation of the predetermined object in the past; a matching means for estimating a current position and orientation of the predetermined object by matching the measured image with a plurality of templates corresponding to different orientations when the past information does not exist or when the past information exists but the estimation means fails to estimate the current position and orientation; a recognition means for recognizing a position and orientation of the predetermined object in the measurement image according to a result of fitting the three-dimensional model to the predetermined object in the measurement image, using a current position and orientation of the predetermined object estimated by either the estimation means or the comparison means as an initial value of a position and orientation of the three-dimensional model; A recognition device comprising:

2. The measuring device further includes a setting unit for setting the measurement range, The estimation means further estimates an estimation range, which is a range in which the predetermined object exists, based on the position or position and orientation estimated by the estimation means; The setting means sets the estimation range as the measurement range.

2. The recognition device according to claim 1 .

3. the estimation means estimates a current position and orientation of the predetermined object based on a change in a position and orientation of the predetermined object between at least two past points in time; 3. The recognition device according to claim 1 or 2.

4. the measurement image represents a result of measurement by an imaging sensor; The position and orientation of the image sensor are controlled by a robot; The estimation means estimates the past position and posture of the predetermined object and the current position and posture of the robot. and estimating a current position and orientation of the predetermined object based on the position and orientation of the predetermined object.

4. The recognition device according to claim 1, wherein the first and second inputs are input to the first and second inputs.

5. The measurement image is a distance image in which each pixel represents a distance to a subject.

5. The recognition device according to claim 1,

6. A recognition device according to any one of claims 1 to 5, an image sensor for measuring the measurement range; a robot that controls a position and orientation of the predetermined object based on the result of recognition by the recognition device; A robot control system comprising:

7. 1. A method for recognizing a position and orientation of a predetermined object in a measurement image by comparing the measurement image, which represents a result of measuring a measurement range including the predetermined object, with a three-dimensional model of the predetermined object, comprising: an estimation step of estimating a current position and orientation of the predetermined object based on past information when past information that is a recognition result of a position and orientation of the predetermined object in the past exists; a matching step of estimating a current position and orientation of the predetermined object by matching the measured image with a plurality of templates corresponding to different orientations when the past information does not exist or when the past information exists but estimation of the current position and orientation has failed in the estimating step; a recognition step of recognizing a position and orientation of the predetermined object in the measurement image according to a result of fitting the three-dimensional model to the predetermined object in the measurement image, using a current position and orientation of the predetermined object estimated in either the estimation step or the matching step as an initial value of a position and orientation of the three-dimensional model; A recognition method comprising the steps of:

8. A program for causing a computer to execute each step of the recognition method according to claim 7.

Images