Method, system, and storage medium for recognizing position and posture of object

Abstract

This application provides a method, system, and computer program for identifying the position and pose of an object, which can accurately identify the position and pose of an object even when the depth cannot be accurately estimated by a depth estimator. The method disclosed herein includes the following steps: (a) generating an object image using a camera; (b) determining the estimated depth of the object using a depth estimator that takes the object image as input and the estimated depth within the object image as output; (c) determining whether the estimated depth is usable using a discriminator that takes at least one of object image features extracted from the object image and the object image as input and the availability of the estimated depth as output; (d) identifying the position and pose of the object using the estimated depth and the object image when the estimated depth is usable; and (e) determining the measured depth of the object using a depth sensor when the estimated depth is unusable and identifying the position and pose of the object using the measured depth.

Description

Technical Field

[0001] This disclosure relates to methods, systems, and storage media for identifying the position and orientation of objects. Background Technology

[0002] A known robotic system uses a camera to identify the position and pose of objects and then performs picking operations. When identifying objects, in many cases, not only conventional images but also depth information are utilized. Patent Document 1 discloses a technique for inferring depth from stereo images and monocular images using a deep learning-based depth inferrer. This technique enables high-speed acquisition of depth from a single captured image.

[0003] Patent Document 1: International Publication No. 2020 / 230214 Summary of the Invention

[0004] However, in the aforementioned prior art, there is a problem that the depth cannot be accurately predicted when an image that is significantly different from the supervised image is input into the depth estimator.

[0005] According to a first aspect of this disclosure, a method for identifying the position and pose of an object is provided, comprising the following steps: (a) generating an image of the object by taking a picture of the object with a camera; (b) determining the estimated depth of the object using a depth estimator that takes the object image as input and an estimated depth within the object image as output; (c) determining whether the estimated depth is usable using a discriminator that takes at least one of object image features extracted from the object image and the object image as input and an output indicating whether the estimated depth is usable; (d) if the estimated depth is usable, identifying the position and pose of the object using the estimated depth and the object image; and (e) if the estimated depth is unusable, determining the measured depth of the object using a depth sensor and identifying the position and pose of the object using the measured depth and the object image.

[0006] According to a second aspect of this disclosure, a system for recognizing the position and pose of an object is provided, comprising: a camera for photographing the object; a depth sensor for measuring the depth of the object; and a recognition unit for performing position and pose recognition processing of the object, the recognition unit performing the following processing: (a) generating an object image by photographing the object using the camera; (b) determining the estimated depth of the object using a depth estimator that takes the object image as input and an estimated depth within the object image as output; (c) determining whether the estimated depth is usable using a discriminator that takes at least one of object image features extracted from the object image and the object image as input and an output indicating whether the estimated depth is usable; (d) recognizing the position and pose of the object using the estimated depth and the object image if the estimated depth is usable; and (e) recognizing the measured depth of the object using the depth sensor and recognizing the position and pose of the object using the measured depth and the object image if the estimated depth is unusable.

[0007] According to a third aspect of this disclosure, a storage medium is provided storing a computer program that causes a processor to perform a process for recognizing the position and pose of an object, the process for recognizing the position and pose of the object comprising the following steps: (a) generating an image of the object by taking a picture of the object with a camera; (b) determining an estimated depth of the object using a depth estimator that takes the object image as input and an estimated depth within the object image as output; (c) determining whether the estimated depth is usable using a discriminator that takes at least one of object image features extracted from the object image and the object image as input and an output indicating whether the estimated depth is usable; (d) recognizing the position and pose of the object using the estimated depth and the object image if the estimated depth is usable; and (e) recognizing the position and pose of the object using a depth sensor if the estimated depth is unusable. Attached Figure Description

[0008] Figure 1 An explanatory diagram illustrating the configuration of the robot system in the embodiment.

[0009] Figure 2 This is a functional block diagram of the information processing device in the embodiment.

[0010] Figure 3 An explanatory diagram illustrating the function of the depth estimator.

[0011] Figure 4 An explanatory diagram illustrating the functions of the feature extraction unit and the discriminator.

[0012] Figure 5 An explanatory diagram illustrating the function of the identification unit.

[0013] Figure 6 This is a flowchart illustrating the entire process of object position and pose recognition.

[0014] Figure 7 A flowchart illustrating the detailed processing flow of step S100 is provided.

[0015] Figure 8A A flowchart illustrating the detailed processing flow of step S200 is provided.

[0016] Figure 8B A flowchart illustrating the detailed processing flow of step S200 is provided.

[0017] Figure 9 A flowchart illustrating the detailed processing flow of step S300 is provided.

[0018] Explanation of reference numerals in the attached figures

[0019] 100: Robot; 110: Base; 120: Robotic arm; 122: Arm end effector; 150: Robotic hand; 200: Control device; 300: Information processing device; 310: Processor; 320: Memory; 330: Interface circuit; 340: Input device; 350: Display device; 410: Camera; 420: Depth sensor; 500: Worktable; 510: First tray; 520: Second tray; 610: Recognition unit; 612: Depth estimator; 614: Feature extraction unit; 616: Discriminator; 618: Position and posture recognition unit; 620: Robot control execution unit. Detailed Implementation

[0020] Figure 1 This is an explanatory diagram illustrating an example of a robot system in one embodiment. The robot system includes a robot 100, a control device 200 for controlling the robot 100, an information processing device 300, a camera 410, a depth sensor 420, and a worktable 500. The information processing device 300 is, for example, a personal computer.

[0021] Robot 100 includes a base 110 and a robotic arm 120. A robotic hand 150, serving as an end effector, is mounted on the end effector 122 at the distal end of the robotic arm 120. The robotic hand 150 can be implemented using a gripper or suction pad capable of holding a workpiece WK. A TCP (Tool Center Point) serving as the control point for the robot 100 is set at the distal end of the robotic hand 150. It should be noted that the control point TCP can be set at any position.

[0022] The robotic arm 120 is composed of six joints J1 to J6 connected sequentially. Of these joints J1 to J6, three joints J2, J3, and J5 are bending joints, and the other three joints J1, J4, and J6 are torsional joints. Although a six-axis robot is illustrated in this embodiment, a robot with any robotic arm mechanism having multiple joints can also be used. Furthermore, while the robot 100 in this embodiment is a vertical multi-joint robot, a horizontal multi-joint robot can also be used.

[0023] A first tray 510 and a second tray 520 are disposed on a worktable 500. Multiple workpieces WK are stored in the first tray 510. The second tray 520 is used as a place to hold the workpieces WK removed from the first tray 510. The robot 100 performs the operation of removing the workpieces WK from the first tray 510 and placing them in the second tray 520. At this time, the workpieces WK are placed in a predetermined position within the second tray 520 with a predetermined posture. To perform this operation accurately, the position and posture of the workpieces WK are identified. The workpieces WK are also referred to as "objects". Although in this embodiment, the position and posture of multiple objects of the same size are identified, the content of this disclosure can also be applied in the case of identifying the position and posture of multiple objects of different sizes.

[0024] A camera 410 is positioned above the first tray 510 to capture images of the workpiece WK within the first tray 510. The images captured by the camera 410 and the depth measured by the depth sensor 420 are used to determine the three-dimensional position and orientation of the workpiece WK. Hereinafter, the three-dimensional position and orientation of the workpiece WK will be referred to as "position and orientation" or "object position and orientation".

[0025] As the camera 410, a stereo camera or a monocular camera can be used, for example. In this embodiment, a stereo RGB camera is used as the camera 410. As described later, a depth estimator can be used to infer the depth from the image captured by the camera 410. As the depth sensor 420, it is preferable to use a sensor that can measure the depth with higher accuracy than the inferred depth obtained using a depth estimator. As such a depth sensor 420, for example, a sensor that utilizes pattern projection methods such as phase shifting or spatial coding can be used. When using pattern projection, a sensor with a projector for projecting a pattern within the field of view is used. It should be noted that an RGBD camera obtained by integrating the camera 410 and the depth sensor 420 can also be used. An RGBD camera is a camera that has an RGB camera for capturing RGB images and a D camera for capturing depth images.

[0026] exist Figure 1The diagram illustrates the world coordinate system Σw, the robot coordinate system Σr, the camera coordinate system Σc, and the sensor coordinate system Σs. The X and Y axes of the world coordinate system Σw are horizontal axes, and the Z axis is the vertical axis. For the other coordinate systems, the reference numerals distinguishing the three coordinate axes are omitted. The robot coordinate system Σr is an orthogonal coordinate system with the predetermined position of the robot 100 as its origin. The camera coordinate system Σc is an orthogonal coordinate system with the predetermined position of the camera 410 as its origin. The sensor coordinate system Σs is an orthogonal coordinate system with the predetermined position of the depth sensor 420 as its origin. The relative relationship between the sensor coordinate system Σs and the camera coordinate system Σc is known; the depth within the field of view of the depth sensor 420 corresponds to the pixel position in the image captured by the camera 410. Furthermore, the camera 410 is pre-calibrated, and the positions and orientations in each coordinate system Σw, Σr, and Σc can be transformed into each other through coordinate transformation.

[0027] Figure 2 This is a block diagram illustrating the function of the information processing device 300. The information processing device 300 includes a processor 310, a memory 320, and an interface circuit 330. An input device 340 and a display device 350 are connected to the interface circuit 330. In addition, a control device 200, a camera 410, and a depth sensor 420 are also connected to the interface circuit 330.

[0028] The processor 310 functions as both an identification unit 610 and a robot control execution unit 620. The identification unit 610 uses a camera 410 and a depth sensor 420 to identify the position and orientation of the workpiece WK. The identification unit 610 includes functions for a depth estimator 612, a feature extraction unit 614, a discriminator 616, and a position and orientation recognition unit 618. The depth estimator 612 takes an image captured by the camera 410 as input to perform depth estimation within the image. The feature extraction unit 614 extracts predetermined image features from the image captured by the camera 410. The discriminator 616 determines whether the estimated depth obtained by the depth estimator 612 can be used for object position and orientation recognition. The position and orientation recognition unit 618 uses the estimated depth or measured depth and an image of the object to perform object position and orientation recognition. The robot control execution unit 620 performs object picking operations using the identified object position and orientation. The functions of the identification unit 610 and the robot control execution unit 620 are implemented by the processor 310 executing computer programs stored in the memory 320. However, some or all of the functions of the identification unit 610 and the robot control execution unit 620 can also be implemented through hardware circuits.

[0029] The memory 320 stores CAD data CD representing the shape of workpiece WK, template TP used in calculating the position and orientation of workpiece WK, first learning data LD1, second learning data LD2, and robot control program RP. The first learning data LD1 includes supervisory data used in the learning process of depth estimator 612. The second learning data LD2 includes supervisory data used in the learning process of discriminator 616. The robot control program RP consists of multiple commands that cause robot 100 to perform actions.

[0030] Figure 3 This is an explanatory diagram illustrating the function of the depth estimator 612. Because the camera 410 in this embodiment is a stereo camera, an object image IM comprising two images M1 and M2 is created. "Object image IM" refers to a photographic image comprising one or more workpieces WK. The depth estimator 612 takes any photographic image captured by the camera 410 as input and outputs the estimated depth De(i,j) within that photographic image. (i,j) represents the two-dimensional coordinates of the estimated depth De. Preferably, the estimated depth De(i,j) has the same resolution as the object image IM, but it may also have a different resolution.

[0031] The depth inferrer 612 can be configured as a machine learning model. For example, the depth inferrer disclosed in Patent Document 1 can be used, or the non-patent document 1 described as prior art in Patent Document 1 (Iro Laina, Christian Rupprecht, Vasileios Belagianis, Federico Tombari, and Nassir Navab, "Deeper Depth Prediction with Fully Convolutional Residual Networks", In Proc. International Conference on 3D Vision (3DV), pp. 239-248, 2016) can be used. Figure 1Publicly available neural networks. Furthermore, the neural network configuration for stereo depth inference described in "End-to-End Learning of Geometry and Context for Deep Stereo Regression" (https: / / arxiv.org / abs / 1703.04309) can also be used. In this configuration, a first network extracts features from the left and right images, and a second network infers disparity, finally converting disparity to depth (https: / / qiita.com / minh33 / items / 1cbcba8f40841997938f). Alternatively, [the following can be used](https: / / qiita.com / minh33 / items / 1cbcba8f40841997938f).

[0032] The neural network described in "Learned Stereo Matching" by omron.com / jp / ja / technology / omrontechnics / 2021 / 20211119-taniai.html

[0033] Figure 4 This is an explanatory diagram illustrating the functions of the feature extraction unit 614 and the discriminator 616. The feature extraction unit 614 extracts object image feature Fo from the object image IM. As the object image feature Fo, for example, an index representing the brightness distribution within the object image IM can be used. The reason for using an index representing the brightness distribution is that it is believed that when the brightness of the object image IM changes significantly due to the influence of external light, the accuracy of the inferred depth obtained from the object image IM will decrease. In this embodiment, a brightness value histogram is used as the object image feature Fo. When N is set to an integer of 2 or more, the brightness value histogram obtained from an RGB image is represented by dividing it into N bins (f1 to f2). N The number of pixels, Np, is obtained by summing the luminance values ​​of the RGB images. Since the object image IM comprises two images, M1 and M2, the object image feature Fo includes 2N features. It should be noted that image features other than the luminance value histogram can also be used as the object image feature Fo. For example, other metrics representing the luminance distribution, such as the mean or variance of the luminance, can be used as the object image feature Fo. Additionally, metrics representing the noise level of the image can also be used as the object image feature Fo. As metrics representing the noise level of the image, for example, edge parameters or multiple spectral components obtained by performing a discrete Fourier transform on the image can be used.

[0034] The discriminator 616 takes at least one of the object image feature Fo and the object image IM as input and outputs a discrimination result DR indicating whether the predicted depth De obtained by the depth inferrer 612 is usable. The discriminator 616 can be configured as a machine learning model. As the machine learning model constituting the discriminator 616, various classification models such as decision trees, support vector machines (SVM), random forests, convolutional neural networks, and logistic regression can be used.

[0035] Figure 5 This is an explanatory diagram illustrating the function of the recognition unit 610. The depth estimator 612 generates an estimated depth De(i,j) from the object image IM(i,j) obtained by the camera 410. On the other hand, the depth sensor 420 generates a measured depth Ds(i,j) by performing depth measurements. Regarding the depth sensor 420, it is preferable to use a sensor capable of obtaining depth with higher accuracy than the depth estimator 612. For example, a depth sensor 420 utilizing pattern projection methods such as phase shifting or spatial coding, although requiring a longer processing time due to multiple shots, can measure depth with high accuracy. On the other hand, for the depth estimator 612, it is preferable to use a depth estimator with a shorter processing time than the depth sensor 420 and capable of obtaining depth at a faster speed. Furthermore, for the depth estimator 612, it is preferable to use a depth estimator capable of obtaining depth with higher accuracy than block matching. Although the estimated depth De(i,j) and the measured depth Ds(i,j) can have different resolutions, in this embodiment, it is assumed that they have information with the same resolution as the object image IM(i,j). The feature extraction unit 614 extracts object image features Fo from the object image IM(i,j). The discriminator 616 takes at least one of the object image features Fo and the object image IM as input and outputs a discrimination result DR indicating whether the predicted depth De(i,j) obtained by the depth inferrer 612 is usable.

[0036] The position and pose recognition unit 618 selectively uses either the inferred depth De(i,j) or the measured depth Ds(i,j) based on the discrimination result DR, and identifies the object's position and pose P(x,y,z,u,v,w). The object image IM is also used in this recognition process. Figure 5 The switch SW depicted indicates the depth selection function of the position and attitude recognition unit 618. It should be noted that when multiple workpieces are included in the field of view of the camera 410, the position and attitude recognition unit 618 preferably recognizes the position and attitude P for at least one of the workpieces.

[0037] The position and pose P(x,y,z,u,v,w) of an object can be represented, for example, by its position (x,y,z) and pose (u,v,w) in a robot coordinate system Σr. The pose (u,v,w) is represented by rotation angles around the three axes. It should be noted that the robot coordinate system Σr can also be replaced by other coordinate systems such as the world coordinate system Σw to represent the object's position and pose. The calculation of the object's position and pose P is performed, for example, using template matching. That is, the object's position and pose are simulated using its CAD data, thereby pre-creating templates TP according to multiple position and poses. The position and pose recognition unit 618 calculates a set of points from the object image IM and the inferred depth De or measured depth Ds, detects the object by performing template matching on the set of points, and identifies or infers its position and pose P. A set of points is a collection of points represented by three-dimensional coordinate values. It should be noted that template matching can also be replaced by other methods to identify the object's position and pose P. For example, machine learning models such as convolutional neural networks can also be used to identify the object's position and pose P.

[0038] Figure 6 This is a flowchart illustrating the entire process of object position and pose recognition during the picking operation of robot 100. In step S100, the recognition unit 610 uses the measured depth Ds measured by the depth sensor 420 to recognize the object's position and pose, and performs learning of the depth inferrer 612. In step S200, the recognition unit 610 uses both the measured depth Ds measured by the depth sensor 420 and the inferred depth De obtained by the depth inferrer 612 to recognize the object's position and pose, and performs learning of the discriminator 616. In step S300, the recognition unit 610 selectively uses either the inferred depth De obtained by the depth inferrer 612 or the measured depth Ds measured by the depth sensor 420 to recognize the object's position and pose based on the discrimination result of the discriminator 616.

[0039] Step S100 is also referred to as the "first learning process," and step S200 is also referred to as the "second learning process." In both the first and second learning processes, it is preferable to collect learning data while the robot 100 is performing object picking operations. This allows the depth inferrer 612 and the discriminator 616 to learn simultaneously while the object picking operation is actually performed. Furthermore, when picking up multiple loose objects, a new scene emerges for recognizing the position and pose of the next object when one object is picked up. Therefore, it is possible to collect learning data sequentially while picking up multiple loose objects one by one. Additionally, since the depth of objects differs in scenes with many objects and scenes with few objects, it is advantageous to create learning data for different depths. The detailed processes of steps S100, S200, and S300 will be described below in sequence.

[0040] Figure 7 A flowchart illustrating the detailed processing flow of step S100 is provided below. In step S101, the recognition unit 610 generates an object image IM by taking a picture of the object using the camera 410. In step S102, the recognition unit 610 obtains a measured depth Ds by performing a measurement using the depth sensor 420. In step S103, the position and pose recognition unit 618 recognizes the position and pose of the object using the measured depth Ds. That is, the position and pose recognition unit 618 recognizes the position and pose of the object using the measured depth Ds and the object image IM. In step S104, the robot control execution unit 620 performs an object picking operation using the object's position and pose. It should be noted that step S104 can also be omitted.

[0041] In step S105, the recognition unit 610 determines whether the first learning data LD1 has been sufficiently accumulated. The first learning data LD1 is the learning data used in the learning of the depth estimator 612, and includes the object image IM and the measured depth Ds. If the first learning data LD1 has not been sufficiently accumulated, the process proceeds to step S106, where the recognition unit 610 saves the dataset including the object image IM and the measured depth Ds as the first learning data LD1 to the memory 320. Then, it returns to step S101 and executes step S101 and subsequent processing again. On the other hand, if the first learning data LD1 has been sufficiently accumulated, the process proceeds to step S107, where the recognition unit 610 uses the first learning data LD1 to perform learning of the depth estimator 612. It should be noted that... Figure 7 The execution order of the steps shown is an example and can be changed as needed. For example, it can be set so that the learning of the depth inferr 612 is performed whenever a new dataset of the first learning data LD1 is obtained. Even in this case, the learning of the depth inferr 612 is equivalent to the processing in step S107 in that it is performed using the first learning data LD1, which includes multiple datasets. When the learning in step S107 ends, the processing in step S100 also ends. It should be noted that the learning in step S107 is preferably performed as a background process. In this case, step S107 can also be performed in parallel with step S200 described later.

[0042] Figure 8A and Figure 8B A flowchart illustrating the detailed processing flow of step S200 is provided. Steps S201 to S204 are... Figure 7Steps S101 to S104 are the same. That is, in step S201, the recognition unit 610 generates an object image IM by taking a picture of the object using the camera 410. In step S202, the recognition unit 610 obtains a measured depth Ds by performing a measurement using the depth sensor 420. In step S203, the position and pose recognition unit 618 uses the measured depth Ds to recognize the position and pose Ps of the object. It should be noted that "position and pose Ps" refers to the position and pose P(x,y,z,u,v,w) recognized using the measured depth Ds. In step S204, the robot control execution unit 620 performs the object picking operation using the object's position and pose Ps. It should be noted that step S204 can also be omitted.

[0043] In step S205, the recognition unit 610 uses the depth estimator 612 to obtain the estimated depth De from the object image IM. In step S206, the position and pose recognition unit 618 uses the estimated depth De to recognize the position and pose Pe of the object. It should be noted that "position and pose Pe" refers to the position and pose P(x,y,z,u,v,w) recognized using the estimated depth De. In step S207, the feature extraction unit 614 extracts the object image feature Fo from the object image IM.

[0044] exist Figure 8B In step S211, the identification unit 610 determines whether the difference between the measured depth Ds and the estimated depth De is below a predetermined value. For example, the difference ΔD between the measured depth Ds and the estimated depth De is calculated using the following formula.

[0045] ΔD=Σ|Ds(i,j)-De(i,j)|: (1)

[0046] Here, Σ illustrates the addition operation performed across the entire range of the two-dimensional coordinates (i,j) of depth Ds.De. It should be noted that, instead of using the value obtained by adding the absolute values ​​of the differences, the difference ΔD can also be obtained by averaging the absolute values ​​of the differences. Alternatively, the Euclidean distance between the measured depth Ds and the inferred depth De can be used as the difference ΔD.

[0047] If the depth difference ΔD is below a predetermined value, it is determined in step S212 that the estimated depth De is usable, and the process proceeds to step S214. On the other hand, if the depth difference ΔD exceeds the predetermined value, it is determined in step S213 that the estimated depth De is unusable, and the process proceeds to step S217, which will be described later.

[0048] In step S214, the identification unit 610 determines whether the difference between the position attitude Ps obtained using the measured depth Ds and the position attitude Pe obtained using the estimated depth De is below a predetermined value. This predetermined value is set independently of the predetermined value used in step S211. Regarding the difference between position attitudes Ps and Pe, for example, each position attitude Ps and Pe can be considered as a six-dimensional vector, and their distance can be calculated. If the difference between position attitudes Ps and Pe is below the predetermined value, in step S215 it is determined that the estimated depth De is usable, and the process proceeds to step S217. On the other hand, if the difference between position attitudes Ps and Pe exceeds the predetermined value, in step S216 it is determined that the estimated depth De is unusable, and the process proceeds to step S217. The processing in steps S211 to S216 is as follows: if the difference in depth ΔD and the difference in position / attitude Ps, Pe are both below their respective predetermined values, the estimated depth De is determined to be usable; if at least one of the differences in depth ΔD and the difference in position / attitude Ps, Pe exceeds a predetermined value, the estimated depth De is determined to be unusable. It should be noted that one of the processing steps S211 to S213 and steps S214 to S216 may be omitted. However, as a direct evaluation of depth error, it is preferable to perform at least steps S211 to S213. The identification unit 610, through the processing in steps S211 to S216, determines a tag indicating whether the estimated depth De is usable based on the difference ΔD between the estimated depth De and the measured depth Ds.

[0049] In step S217, the recognition unit 610 determines whether the second learning data LD2 has been sufficiently accumulated. The second learning data LD2 is the learning data used in the learning of the discriminator 616, and includes at least one of the object image IM and the object image feature quantity Fo, as well as a label indicating whether the predicted depth De is usable. However, if the object image IM is not used as input to the discriminator 616, it is not necessary to save the object image IM as the second learning data LD2. If the object image feature quantity Fo is not used as input to the discriminator 616, the object image feature quantity Fo is not needed, and step S207 described above can also be omitted.

[0050] Without sufficient accumulation of the second learning data LD2, the process proceeds to step S218, where the recognition unit 610 saves a dataset including at least one of the object image IM and the object image feature Fo, along with a label indicating whether the predicted depth De is usable, as the second learning data LD2 to the memory 320. Then, it returns to... Figure 8AStep S201 is executed again, along with subsequent processing. On the other hand, after sufficient accumulation of the second learning data LD2, the process proceeds to step S219, where the recognition unit 610 uses the second learning data LD2 to perform learning for the discriminator 616. It should be noted that... Figure 8A and Figure 8B The execution order of the steps shown is an example and can be changed as needed. For example, it can be set so that the discriminator 616 learns whenever a new dataset of the second learning data LD2 is obtained. Even in this case, the learning of the discriminator 616 is equivalent to the processing in step S219 in that it uses the second learning data LD2, which includes multiple datasets. When the learning in step S219 ends, the processing in step S200 also ends. It should be noted that steps S205-S207 and S211-S219 are preferably executed as background processes.

[0051] Figure 9 A flowchart illustrating the detailed processing flow of step S300 is provided. Figure 9 The processing and in Figure 5 The processing described herein is the same. In step S301, the recognition unit 610 generates an object image IM by taking a picture of the object using the camera 410. In step S302, the recognition unit 610 uses the depth estimator 612 to obtain the estimated depth De from the object image IM. In step S303, the feature extraction unit 614 extracts the object image feature Fo from the object image IM. In step S304, the recognition unit 610 uses the discriminator 616 to determine whether the estimated depth De is usable from at least one of the object image feature Fo and the object image IM.

[0052] If the inferred depth De is available, the process proceeds from step S305 to step S306, where the position and pose recognition unit 618 uses the inferred depth De to identify the position and pose of the object. Conversely, if the inferred depth De is unavailable, the process proceeds from step S305 to step S307, where the recognition unit 610 uses the depth sensor 420 to acquire the measured depth Ds. In step S308, the position and pose recognition unit 618 uses the measured depth Ds to identify the position and pose of the object. It should be noted that, as in... Figure 5 As explained, the object image IM is also used in position and pose recognition. In step S309, the robot control execution unit 620 performs the object picking operation using the object's position and pose. It should be noted that the processing time required for steps S302 to S304 is preferably shorter than the time required for depth measurement in step S307. In this way, position and pose recognition processing can be performed efficiently using the inferred depth De. It should be noted that... Figure 9 The execution order of the steps shown is an example and can be changed as needed.

[0053] In summary, in the above embodiments, since the estimated depth De is used to identify the object's position and pose when the discriminator 616 determines that the estimated depth De is usable, the object's position and pose can be identified with sufficiently high accuracy and high speed. Furthermore, since the measured depth Ds measured by the depth sensor 420 is used to identify the object's position and pose when the estimated depth De is not usable, the position and pose can be identified with high accuracy. It should be noted that the content of this disclosure can also be applied to the processing of identifying the position and pose of objects in devices other than robots.

[0054] Other methods:

[0055] This disclosure is not limited to the embodiments described above, and can be implemented in various ways without departing from its spirit. For example, this disclosure can also be implemented in the following aspects. In order to solve part or all of the technical problems of this disclosure, or to achieve part or all of the effects of this disclosure, the technical features in the above embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined. In addition, technical features that are not described as necessary in this specification can be appropriately deleted.

[0056] (1) According to a first aspect of this disclosure, a method for identifying the position and pose of an object is provided. The method includes the steps of: (a) generating an image of the object by taking a picture of the object using a camera; (b) determining the estimated depth of the object using a depth estimator that takes the object image as input and an estimated depth within the object image as output; (c) determining whether the estimated depth is usable using a discriminator that takes at least one of object image features extracted from the object image and the object image as input and an output indicating whether the estimated depth is usable; (d) if the estimated depth is usable, identifying the position and pose of the object using the estimated depth and the object image; and (e) if the estimated depth is unusable, determining the measured depth of the object using a depth sensor and identifying the position and pose of the object using the measured depth and the object image.

[0057] According to this method, when it is determined that the predicted depth obtained using a depth estimator is usable, using this predicted depth allows for high-speed identification of the object's position and orientation with sufficiently high accuracy. Furthermore, when it is determined that the predicted depth is unusable, the object's position and orientation are identified using the measured depth obtained by a depth sensor, thus enabling high-precision identification of the position and orientation.

[0058] (2) In the above method, it can also be set that the processing of using the depth estimator to calculate the estimated depth is less accurate and faster than the processing of using the depth sensor to calculate the measured depth.

[0059] According to this method, the features of depth inferrers and depth sensors can be flexibly used to identify the position and pose of objects with sufficiently high accuracy and speed.

[0060] (3) In the above method, it can also be set that before the steps (a) to (e), the method includes: a first learning step, to learn the depth inferrer; and a second learning step, to learn the discriminator. The first learning step and the second learning step each include a step of collecting learning data while performing the robot's picking up operation on the object according to the position and posture of the object.

[0061] According to this method, it is possible to perform the learning of depth inferrers and discriminators while actually picking up objects.

[0062] (4) In the above method, the first learning step may also be configured to include the following steps: (1a) generating a first learning image by taking a picture of the object using the camera; (1b) determining a first learning measurement depth of the object using the depth sensor; (1c) recognizing the position and pose of the object using the first learning measurement depth and the first learning image and performing the robot's picking-up operation on the object; (1d) generating multiple first learning data, each including the first learning image and the first learning measurement depth, by executing steps (1a) to (1c) multiple times; and (1e) performing the learning of the depth inferr using the multiple first learning data.

[0063] According to this method, the depth inferrer can be learned reasonably while the object picking operation is being performed.

[0064] (5) In the above method, the second learning step may also be configured to include the following steps: (2a) generating a second learning image by taking a picture of the object using the camera; (2b) determining a second learning inferred depth from the second learning image using the depth inferred generator; (2c) determining a second learning measured depth of the object using the depth sensor; (2d) recognizing the position and pose of the object using the second learning measured depth and the second learning image and performing the robot's picking-up operation on the object; (2e) based on the second learning... (2f) A step of determining whether a label representing the second learning depth can be used by using the difference between the inferred depth and the second learning measured depth; (2g) A step of extracting the learning image features of the second learning image when the discriminator is configured to take the object image features as input; (2f) A step of generating a plurality of second learning data, each including the label and at least one of the learning image features and the second learning image, by repeatedly executing the steps (2a) to (2f); and (2h) A step of performing the learning of the discriminator using the plurality of second learning data.

[0065] According to this method, object picking can be performed while the discriminator learns in a reasonable manner.

[0066] (6) According to a second aspect of this disclosure, a system for identifying the position and pose of an object is provided. The system includes: a camera for photographing the object; a depth sensor for measuring the depth of the object; and an identification unit for performing position and pose identification processing of the object. The identification unit performs the following processing: (a) generating an image of the object by photographing it with the camera; (b) determining the estimated depth of the object using a depth estimator that takes the object image as input and outputs an estimated depth within the object image; (c) determining whether the estimated depth is usable using a discriminator that takes at least one of object image features extracted from the object image and the object image as input and outputs whether the estimated depth is usable; (d) if the estimated depth is usable, identifying the position and pose of the object using the estimated depth and the object image; and (e) if the estimated depth is unusable, determining the measured depth of the object using the depth sensor and identifying the position and pose of the object using the measured depth and the object image.

[0067] (7) According to a third aspect of this disclosure, a computer program is provided that causes a processor to perform processing to identify the position and pose of an object. The computer program causes the processor to perform the following processing: (a) generating an image of the object by taking a picture of the object using a camera; (b) determining the estimated depth of the object using a depth estimator that takes the object image as input and an estimated depth within the object image as output; (c) determining whether the estimated depth is usable using a discriminator that takes at least one of object image features extracted from the object image and the object image as input and an output indicating whether the estimated depth is usable; (d) identifying the position and pose of the object using the estimated depth and the object image if the estimated depth is usable; and (e) determining the measured depth of the object using a depth sensor and identifying the position and pose of the object using the measured depth and the object image if the estimated depth is unusable.

[0068] This disclosure can also be implemented in various ways other than those described above. For example, it can be implemented by a robot system having a robot and a robot control device, a computer program for implementing the functions of the robot control device, or a non-transitory storage medium on which the computer program is recorded.

Claims

1. A method of identifying a position and orientation of an object, characterized by, The method includes the following processes: (a) generating an object image by taking a picture of the object using a camera; (b) calculating a presumed depth of the object using a depth presumer that takes the object image as input and outputs a presumed depth within the object image; (c) determining whether the presumed depth can be used using a discriminator that takes at least one of an object image feature quantity extracted from the object image and the object image as input and outputs whether the presumed depth can be used; (d) in the case where the presumed depth can be used, recognizing a position and posture of the object using the presumed depth and the object image; and (e) in the case where the presumed depth cannot be used, calculating a measured depth of the object using a depth sensor and recognizing a position and posture of the object using the measured depth and the object image, Before the processes (a) to (e), the method includes a first learning process of performing learning of the depth presumer and a second learning process of performing learning of the discriminator, The first learning process and the second learning process each include a process of collecting learning data while a robot performs a picking operation on the object according to a position and posture of the object, The second learning process includes the following processes: (2a) generating a second learning image by taking a picture of the object using the camera; (2b) calculating a second learning presumed depth from the second learning image using the depth presumer; (2c) calculating a second learning measured depth of the object using the depth sensor; (2d) recognizing a position and posture of the object using the second learning measured depth and the second learning image and performing a picking operation on the object by the robot; (2e) determining a label indicating whether the second learning presumed depth can be used according to a difference between the second learning presumed depth and the second learning measured depth; (2f) in the case where the discriminator is configured to take the object image feature quantity as input, extracting a learning image feature quantity of the second learning image; (2g) generating a plurality of second learning data each including at least one of the learning image feature quantity and the second learning image and the label by repeatedly performing the processes (2a) to (2f); and (2h) performing learning of the discriminator using the plurality of second learning data.

2. The method of recognizing a position and posture of an object according to claim 1, wherein The process of calculating the presumed depth using the depth presumer is less accurate and faster than the process of calculating the measured depth using the depth sensor.

3. The method of recognizing a position and posture of an object according to claim 1, wherein The first learning process includes the following processes: (1a) generating a first learning image by taking a picture of the object using the camera; (1b) calculating a first learning measured depth of the object using the depth sensor; ​ (1c) recognizing a position and posture of the object using the first learning-use measured depth and the first learning-use image, and performing a picking operation of the robot on the object; (1d) generating a plurality of first learning-use data each including the first learning-use image and the first learning-use measured depth by repeatedly performing the processes (1a) to (1c); and (1e) performing learning of the depth estimator using the plurality of first learning-use data.

4. A system for identifying a position and orientation of an object, characterized by provided with: a camera that captures the object; a depth sensor that measures a depth of the object; and an identification unit that performs an identification process of a position and posture of the object, the identification unit performs the following processes: (a) generating an object image by capturing the object using the camera; (b) calculating a predicted depth of the object using a depth estimator that takes the object image as input and outputs a predicted depth within the object image; (c) determining whether the predicted depth can be used using a discriminator that takes at least one of an object image feature quantity extracted from the object image and the object image as input and outputs whether the predicted depth can be used; (d) in a case where the predicted depth can be used, recognizing a position and posture of the object using the predicted depth and the object image; and (e) in a case where the predicted depth cannot be used, calculating a measured depth of the object using the depth sensor and recognizing a position and posture of the object using the measured depth and the object image, before the processes (a) to (e), further including a first learning process of performing learning of the depth estimator and a second learning process of performing learning of the discriminator, the first learning process and the second learning process each include a process of collecting learning-use data while performing a picking operation of a robot on the object according to a position and posture of the object, the second learning process includes the following processes: (2a) generating a second learning-use image by capturing the object using the camera; (2b) calculating a second learning-use predicted depth from the second learning-use image using the depth estimator; (2c) calculating a second learning-use measured depth of the object using the depth sensor; (2d) recognizing a position and posture of the object using the second learning-use measured depth and the second learning-use image, and performing a picking operation of the robot on the object; (2e) determining a label indicating whether the second learning-use predicted depth can be used according to a difference between the second learning-use predicted depth and the second learning-use measured depth; (2f) in a case where the discriminator is configured to take the object image feature quantity as input, extracting a learning-use image feature quantity of the second learning-use image; (2g) generating a plurality of second learning-use data each including at least one of the learning-use image feature quantity and the second learning-use image and the label by repeatedly performing the processes (2a) to (2f); and (2h) performing learning of the discriminator using the plurality of second learning-use data.

5. A storage medium, characterized by The storage medium stores a computer program, The computer program is a computer program that causes a processor to execute a process of recognizing a position and posture of an object, the process of recognizing the position and posture of the object including processes of: (a) generating an object image by capturing the object using a camera; (b) calculating a presumed depth of the object using a depth presumer that takes the object image as input and outputs a presumed depth within the object image; (c) determining whether the presumed depth can be used using a discriminator that takes at least one of an object image feature amount extracted from the object image and the object image as input and outputs whether the presumed depth can be used; (d) recognizing the position and posture of the object using the presumed depth and the object image when the presumed depth can be used; and (e) calculating a measured depth of the object using a depth sensor when the presumed depth cannot be used and recognizing the position and posture of the object using the measured depth and the object image, before the processes (a) to (e), further including a first learning process of performing learning of the depth presumer and a second learning process of performing learning of the discriminator, the first learning process and the second learning process each including a process of collecting learning data while a robot performs a picking operation on the object based on the position and posture of the object, the second learning process including processes of: (2a) generating a second learning image by capturing the object using the camera; (2b) calculating a second learning presumed depth from the second learning image using the depth presumer; (2c) calculating a second learning measured depth of the object using the depth sensor; (2d) recognizing the position and posture of the object using the second learning measured depth and the second learning image and performing the picking operation on the object by the robot; (2e) determining a label indicating whether the second learning presumed depth can be used based on a difference between the second learning presumed depth and the second learning measured depth; (2f) extracting a learning image feature amount of the second learning image when the discriminator is configured to take the object image feature amount as input; (2g) generating a plurality of second learning data each including at least one of the learning image feature amount and the second learning image and the label by repeatedly performing the processes (2a) to (2f); and (2h) performing learning of the discriminator using the plurality of second learning data. ​

Images