Method and device for selecting a six-dimensional pose hypothesis from multiple six-dimensional pose hypotheses
Through parallel computing and iterative methods, the problem of slow rotational symmetry object recognition speed and high calculation cost in machine vision system is solved, and fast and accurate object recognition is achieved .
Patent Information
- Application Number
- CN202080000792.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-04-14
- Filing Date
- 2020-04-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-04-23
AI Technical Summary
When identifying objects with rotational symmetry or large planes, machine vision systems face the problem of high recognition speed, accuracy and computational cost, especially when selecting candidate poses from multiple six-dimensional pose assumptions, the prior art cannot Fast and accurate implementation.
By applying mathematical operations in parallel, the model three-dimensional point cloud data is extracted from multiple six-dimensional pose assumptions, translated and rotated, and multiple transformed model three-dimensional point cloud data instances are generated, and the overlap with the target three-dimensional point cloud data is determined through iterative operations. Degree, select the six-dimensional pose assumption with the largest overlap coefficient as the candidate pose.
It realizes the rapid and accurate identification of candidate poses from multiple six-dimensional pose assumptions, improves the speed and accuracy of object recognition, reduces the calculation cost, and is suitable for object recognition tasks in computer vision systems.
Smart Images

Figure CN112352262B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to object recognition, and in particular to identifying an object based on selecting a six-dimensional pose hypothesis from multiple six-dimensional pose hypotheses. Background Art
[0002] It does not seem difficult for humans to quickly, accurately, and precisely identify a desired object from among many others and then grasp the desired object. The precise means by which this routine activity is achieved is not well understood. It is a challenge to replicate the speed, precision, and accuracy of human object recognition in machine vision systems.
[0003] In a machine vision system, an imaging device, such as a camera or scanner, generates an image of multiple objects from which a particular object is to be identified. A machine vision system may include a database of three-dimensional representations of objects that the machine may frequently encounter. For example, if a machine vision system is integrated into a robot, if that robot is deployed in an automobile factory, the machine vision system is likely to encounter automobile parts such as doors, springs, gears, and pipes. Therefore, such a machine vision system may include a three-dimensional representation of a pipe in its memory.
[0004] A machine vision system (e.g., under the control of a processor of the machine vision system) will attempt to match the three-dimensional representation of the object with an image of a particular object to be identified. However, one difficulty associated with such matching is that the particular object to be selected may be randomly located (e.g., located at a particular position of the axes of a Cartesian coordinate system) and oriented (e.g., at a particular Euler angle within the Cartesian coordinate system) within a container of objects of similar size and shape. Furthermore, the known coordinate system of the imaging device of the machine vision system may be different from the coordinate system of the object to be identified, thereby requiring a conversion from the coordinate system of the object to the coordinate system of the imaging device. Therefore, in order to accurately match the image of a particular object with the three-dimensional representation of the object, the machine vision system may generate three-dimensional representations of the object at different positions and orientations. These three-dimensional representations of the object at different positions and orientations are often referred to as six-dimensional pose hypotheses.
[0005] A machine vision system may generate thousands of six-dimensional pose hypotheses, and the machine vision system may attempt to match each six-dimensional pose hypothesis with an image of a particular object to be identified. Although this brute force approach may eventually lead to a correct match between the six-dimensional pose hypothesis and the image of the particular object to be identified, this brute force approach is computationally expensive (i.e., uses a lot of processing time) and is typically slow because the machine vision system tests each six-dimensional pose hypothesis in turn until a correct six-dimensional pose hypothesis is identified. In most practical applications, slow solutions are unacceptable. Computationally expensive solutions are also unacceptable in most practical applications because it may not be easy to have sufficient computing resources to implement these types of solutions. Summary of the invention
[0006] Due to the inherent difficulties in accurately, precisely and quickly identifying objects with rotational symmetry or objects with large flat surfaces, there is a need to enhance the identification of a candidate six-dimensional pose hypothesis from multiple six-dimensional pose hypotheses without the aid of edge matching techniques or other techniques known in the art that are unable to accurately, precisely and quickly identify such objects. The candidate six-dimensional pose hypothesis corresponds to translating and rotating the three-dimensional representation of the object, positioning and orienting the three-dimensional representation of the object so that the position and orientation of the three-dimensional representation of the object are substantially consistent with an image of the object at a random position and orientation. Multiple six-dimensional pose hypotheses correspond to assumed translations and rotations of the three-dimensional representation of the object, many of which are not substantially consistent with the image of the object at a random position and orientation.
[0007] The technical embodiments described herein solve the problem of identifying an object from images of multiple objects containing the image of the object. In a specific embodiment, the image can be a grayscale image with a depth image, or a red, green, and blue (RGB) image with a depth image. According to an embodiment of the present invention, the object of the recognition processing provided may have, for example, rotational symmetry, a large plane, and / or one or more other features that are generally problematic with respect to machine-based recognition processing. The solution provided herein provides a method for quickly, accurately and precisely identifying candidate six-dimensional pose hypotheses from multiple six-dimensional pose hypotheses. According to an embodiment of the present invention, the translation and rotation corresponding to the candidate six-dimensional pose hypothesis can basically correctly orient and position the three-dimensional representation of the object so as to establish a match between the three-dimensional representation of the object and the image of the object. In this way, the embodiments described herein can be deployed in a computer vision system to help quickly, accurately and precisely identify an object among multiple objects.
[0008] Embodiments of the present invention include a method for rapidly identifying a candidate six-dimensional pose hypothesis from a plurality of six-dimensional pose hypotheses. Model three-dimensional point cloud data extracted from a three-dimensional representation of an object geometrically similar to an object to be identified from an image of a plurality of objects containing the object can be transformed by selecting a first set of six-dimensional pose hypotheses from the plurality of six-dimensional pose hypotheses in parallel.
[0009] Each six-dimensional pose hypothesis in the first group of six-dimensional pose hypotheses can correspond to a specific translation and rotation matrix. When the coordinates of the model three-dimensional points of the model three-dimensional point cloud data are operated by specific translation and rotation matrices, for example, these model three-dimensional points can be translated and rotated in the coordinate system of an imaging device used to generate images of multiple objects. These translated and rotated model three-dimensional points include instances of converted model three-dimensional point cloud data according to an embodiment. Further, through the mathematical operations described herein, multiple instances of converted model three-dimensional point cloud data can be generated in parallel. Through parallelized mathematical operations, candidate six-dimensional pose hypotheses corresponding to the position and orientation of the object to be identified can be quickly generated.
[0010] The model three-dimensional point cloud data can be iteratively operated by selecting one or more subsequent groups of six-dimensional pose hypotheses (for example, a second group of six-dimensional pose hypotheses, a third group of six-dimensional pose hypotheses, etc. suitable for any specific implementation) from multiple six-dimensional pose hypotheses. In this way, multiple instances of multiple groups of converted model three-dimensional point cloud data can be quickly generated. The degree of overlap corresponding to the image of the object to be identified can be determined between the model three-dimensional points of the converted model three-dimensional point cloud data and the three-dimensional points of the target three-dimensional point cloud data. The six-dimensional pose hypothesis corresponding to the maximum number of converted model three-dimensional points associated with the instance of the converted model three-dimensional point cloud data that coincides with the target three-dimensional points in the target three-dimensional point cloud data can be determined as a candidate six-dimensional pose hypothesis (that is, a candidate six-dimensional pose hypothesis is selected from multiple six-dimensional pose hypotheses as a pose hypothesis representing the pose of the object in the image).
[0011] The candidate six-dimensional pose hypotheses determined according to the concepts of the present invention can be used in a variety of ways for various applications. For example, candidate six-dimensional pose hypothesis data corresponding to the candidate six-dimensional pose hypothesis can be provided to a robot for identifying an object to be identified from an image of multiple objects including the object to be identified. The robot can grasp the object to be identified based at least in part on the candidate six-dimensional pose hypothesis data. As another example, the candidate six-dimensional pose hypothesis data can be used to identify a specific molecule with a specific geometric shape from an image of molecules with multiple different orientations and positionings. In this way, drug discovery can be accelerated because the pharmacological effects of many drugs depend on the geometric shape of the molecules.
[0012] The features and technical advantages of the present invention have been outlined quite extensively above so that the detailed description of the invention that follows may be better understood. Other features and advantages of the present invention that form the subject of the claims of the present invention will be described below. It should be understood by those skilled in the art that the disclosed concepts and specific embodiments may be easily used as a basis for modifying or designing other structures to achieve the same purposes of the present invention. It should also be recognized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the present invention as set forth in the appended claims. The novel features that are believed to be characteristic of the present invention, in terms of its organization and method of operation, as well as other objects and advantages, will be better understood by considering the following description in conjunction with the accompanying drawings. However, it should be expressly understood that each of the accompanying drawings is provided for the purpose of illustration and description only and is not intended to be a definition of limitations of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more fully understand the present disclosure, reference is now made to the following description in conjunction with the accompanying drawings, in which:
[0014] Figure 1A and 1B A system for identifying an object from multiple objects according to an embodiment of the present invention is shown.
[0015] Figure 2A-1 , 2B-1 , 2A-2 and 2B-2 describe an overview of an exemplary method for selecting a candidate six-dimensional pose hypothesis from multiple six-dimensional pose hypotheses according to an embodiment of the present invention.
[0016] Figure 3 It is a detailed flowchart of an exemplary method for selecting a candidate six-dimensional pose hypothesis from multiple six-dimensional pose hypotheses according to an embodiment of the present invention.
[0017] Figure 4A and Figure 4B 2A, 2B and Figure 3 An exemplary data structure for the exemplary method described in .
[0018] Figure 5A and 5B Depicting iterating a second set of six-dimensional pose hypotheses using a data structure according to an embodiment of the present invention;
[0019] Fig. 6A and 6B An exemplary method for generating multiple six-dimensional pose hypotheses according to an embodiment of the present invention is described; and
[0020] Figure 7 4 is a flowchart of an exemplary method for improving candidate pose hypotheses according to an embodiment of the present invention. DETAILED DESCRIPTION
[0021] Specific embodiments of the present invention will be described below with reference to the accompanying drawings. In the description, common features are designated by common reference numerals throughout the accompanying drawings. As used herein, ordinal terms (such as "first", "second", "third", etc.) used to modify elements such as structures, components, operations, etc. do not themselves indicate any priority or order of the element relative to another element, but are merely used to distinguish the element from another element with the same name (but using ordinal terms).
[0022] As used herein, various terms are used only to describe particular embodiments and are not intended to limit the embodiments. For example, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and "comprising" can be used interchangeably with "includes" or "including." In addition, it will be understood that the term "wherein" can be used interchangeably with "where."
[0023] Figure 1A and 1B A system 100 configured to identify a particular object 102 from a plurality of objects 104 in accordance with the concepts of the present disclosure is described. For example, implementations of the system 100 in accordance with the illustrated embodiment may be used to facilitate rapid, precise, and / or accurate identification of objects in a variety of scenarios. For example, the system 100 may be used to provide identification of objects having features that are typically problematic with respect to machine-based recognition processing. In some embodiments, the system 100 may quickly, precisely, and accurately identify objects having rotational symmetry or large flat surfaces. However, the concepts of the present disclosure are not limited to identifying such objects. In particular, embodiments implemented in accordance with the concepts of the present disclosure may identify objects of various structures by correctly identifying a candidate six-dimensional pose hypothesis from a plurality of six-dimensional pose hypotheses, as described in detail below.
[0024] The plurality of objects 104 (from which the system 100 can identify a specific object) may include a plurality of objects of random positioning, orientation, size, and shape, including one or more specific objects of interest (shown as specific object 102), which are placed in a container 106 or otherwise intermixed. According to some embodiments, the specific object 102 may be a shape lacking distinct edges, such as a shape consisting of two substantially attached cylinders (as shown in the figure) or another shape lacking distinct edges (such as a sphere). In addition, or alternatively, the specific object 102 may be a shape having a large flat surface, such as object 108 in the plurality of objects 104.
[0025] The system 100 of the illustrated embodiment may include: Figure 1B The computing device 110, robot 112, imaging device 114, and network 120 are shown. For example, the imaging device 114 may correspond to a camera or a scanner. The imaging device 114 of the present embodiment may be configured to generate an image 116 of the plurality of objects 104. The image 116 may correspond to a three-dimensional image of an object, such as a three-dimensional image of a particular object 102 or a plurality of objects 104. As an example, the image 116 may be a grayscale image with a depth image, or the image 116 may be a red, green, and blue (RGB) image with a depth image. The imaging device 114 may be a time-of-flight camera capable of generating the image 116, which may be a three-dimensional image. Alternatively, the imaging device 114 may include at least two stereo positioning cameras, such that a first image generated by a first stereo positioning camera and a second image generated by a second stereo positioning camera may be combined to generate a three-dimensional image. The image 116 may include information about the position of the plurality of objects 104 and the orientation of the plurality of objects 104. In particular, the image 116 may include six-dimensional pose data of the six-dimensional pose of the plurality of objects 104 relative to a known coordinate system of the imaging device 114. The six-dimensional pose of the object may include information about the position of the object in the three-dimensional coordinate system (e.g., the x, y, and z axes of a Cartesian coordinate system corresponding to the field of view of the imaging device (e.g., the imaging device 114)) and the orientation of the object in the three-dimensional coordinate system (e.g., the Euler angles α, β, and γ of a Cartesian coordinate system corresponding to the field of view of the imaging device (e.g., the imaging device 114)). The imaging device 114 may include functionality such as a transmitter (not depicted) to transmit the image 116 to the computing device 110 via the network 120.
[0026] like Figure 1B The computing device 110 shown may include a processor 122, a memory 124, an input / output device 126, and a network interface 128, all of which may be Figure 1B As shown. Figure 1B As shown, memory 124 may be a random access memory (RAM), a dynamic random access memory (DRAM), or other electronic memory. Memory 124 may include a ROM device, a RAM device, one or more HDDs, flash memory devices, an SSD, other devices configured to store data in a persistent or non-persistent state, or a combination of different memory devices. Memory 124 may include one or more memory elements. Memory 124 may be configured to store a database of three-dimensional representations of objects, such as computer-aided design (CAD) representations of objects, similar to Figure 1A A specific object depicted in 102, or Figure 1A As an example, in Figure 1BThe specific three-dimensional representation of the object 130 depicted in FIG. 1 is geometrically similar to Figure 1A 102 is similar to the specific object 102 shown in . In another embodiment, the memory 124 can be configured to store a three-dimensional representation of an object, such as a three-dimensional representation of an object 130 in an Initial Graphics Exchange Specification (IGES) format, a Drawing eXchange Format (DXF), or other suitable graphic data storage format. The memory 124 can also be configured to store instructions that, when executed by the processor 122, cause the processor 122 to perform the exemplary method described in the present invention. In addition, the memory 124 can be configured to store target three-dimensional point cloud data 132 extracted from the image 116 and other data, such as Figure 1B shown.
[0027] Figure 1B The processor 122 depicted in the figure may be a graphics processing unit (GPU) or other processor known in the art capable of processing image data. For example, according to various aspects described herein, the processor 122 may include various forms of processor-based systems. The processor 122 may include a general-purpose computer system (such as a personal computer (PC), a server, a tablet device, etc.) and / or a dedicated processor platform (such as an application-specific integrated circuit (ASIC), a system on a chip (SoC), etc.). In some embodiments, the processor 122 may include one or more processors. The processor 122 may be configured to Figure 1A The three-dimensional point cloud data may encode the position (relative to the imaging device 114) and the direction (relative to the imaging device 114) of each pixel of the image 116 into a data format that can be stored in the memory 124. The three-dimensional point cloud data corresponding to the image 116 may be referred to as the target three-dimensional point cloud data 132, such as Figure 1B The processor 122 may also be configured to store the target three-dimensional point cloud data 132 in the memory 124. In addition, the processor 122 may be configured to convert the three-dimensional representation of the object 130 into three-dimensional point cloud data, such as Figure 1B As shown, it is referred to as model three-dimensional point cloud data 134 . In addition, the processor 122 may be configured to store the model three-dimensional point cloud data 134 in the memory 124 .
[0028] In addition, the processor 122 may be configured to generate six-dimensional pose hypothesis data 136, such as Figure 1BAs shown, it corresponds to multiple six-dimensional pose hypotheses. Each of the multiple six-dimensional pose hypotheses can correspond to a specific translation matrix and a specific rotation matrix. When the coordinates of a model three-dimensional point of the model three-dimensional point cloud data 134 are operated (such as multiplied) with the translation matrix, the coordinates of the model three-dimensional point can be transformed in the spatial direction of the coordinate system of the imaging device 114. When the coordinates of the model three-dimensional point of the model three-dimensional point cloud data 134 are operated (such as multiplied) with the rotation matrix, the coordinates of the model three-dimensional point can be transformed along the Euler angle of the coordinate system of the imaging device 114. The processor 122 can be configured to generate six-dimensional pose hypothesis data 136 by applying a local three-dimensional point pair feature matching technique known in the art, an illustrative example of which will be described in detail in detail in FIG. Fig. 6A and 6B Alternatively, other known techniques are used to generate the six-dimensional pose hypothesis data 136. For example, a random sampling consensus technique may be used to generate the six-dimensional pose hypothesis data 136. The processor 122 may also be configured to store the six-dimensional pose hypothesis data 136 in the memory 124.
[0029] In addition, the processor 122 may be configured to select a candidate six-dimensional pose hypothesis data from the six-dimensional pose hypothesis data 136. In one embodiment, the processor 122 may be configured to select a first data corresponding to a first group of six-dimensional pose hypotheses from the six-dimensional pose hypothesis data 136. The first data corresponding to the first group of six-dimensional pose hypotheses may be randomly selected from the six-dimensional pose hypothesis data 136. Optionally, the processor 122 may be configured to assign a value to each of the multiple six-dimensional pose hypotheses. The value may indicate the probability that the six-dimensional pose hypothesis corresponds to the candidate six-dimensional pose hypothesis.
[0030] In one embodiment, the processor 122 may be configured to perform parallel mathematical operations on an instance of the model 3D point cloud data 134 using first data corresponding to the first set of six-dimensional pose hypotheses to generate a transformed model 3D point cloud data instance. Each transformed model 3D point cloud data instance may correspond to a model 3D point of the model 3D point cloud data 134 that is translated and rotated in a specific and unique manner by a six-dimensional pose hypothesis selected from the first set of six-dimensional pose hypotheses. For example, the processor 122 may multiply the coordinates of an instance of the model 3D point cloud data 134 by the translation and rotation matrices corresponding to each six-dimensional pose hypothesis of the first set of six-dimensional pose hypotheses in parallel to generate transformed model 3D point cloud data for each unique instance.
[0031] In another embodiment, the processor 122 may be configured to generate a threshold number of copies of the model three-dimensional point cloud data 134. The threshold number of copies of the model three-dimensional point cloud data 134 may correspond to the size of the memory 124. The processor 122 may be configured to store data corresponding to each coordinate of each point of the model three-dimensional point cloud data 134 at an address of the memory 124. The processor 122 may be configured to perform parallel mathematical operations on each copy of the model three-dimensional point cloud data using each six-dimensional pose hypothesis in the first plurality of six-dimensional pose hypotheses to generate a plurality of instances of transformed model three-dimensional point cloud data. For illustration, the processor 122 may be configured to multiply each coordinate of each model three-dimensional point of each copy of the model three-dimensional point cloud data in parallel by the translation and rotation matrices corresponding to each six-dimensional pose hypothesis of the first plurality of six-dimensional pose hypotheses. The output of the aforementioned operation may include a plurality of instances of transformed model three-dimensional point cloud data.
[0032] In an embodiment of the present invention, the plurality of six-dimensional pose hypotheses may exceed the first set of six-dimensional pose hypotheses. Therefore, the processor 122 may be configured to iteratively process a second set of six-dimensional pose hypotheses selected from the six-dimensional pose hypothesis data 136 in the same manner as described above. For example, the processor 122 may be configured to perform parallel mathematical operations on an instance of the model three-dimensional point cloud data 134 using second data corresponding to a second set of six-dimensional pose hypotheses selected from the six-dimensional pose hypothesis data 136 to generate an instance of the transformed model three-dimensional point cloud data.
[0033] The processor 122 may also be configured to determine an overlap between the transformed model 3D points (which correspond to an instance of the transformed model 3D point cloud data) and the target 3D point cloud data 132. In addition, the processor 122 may be configured to determine a 6D pose hypothesis (e.g., a first 6D pose hypothesis in the first set of 6D pose hypotheses or a second 6D pose hypothesis in the second set of 6D pose hypotheses) that produces a maximum number of transformed model 3D points that overlap with the target 3D points. The 6D pose hypothesis that produces the maximum number of transformed 3D points that overlap with the target 3D points may correspond to a 6D pose hypothesis of a candidate pose.
[0034] In one embodiment, the processor 122 may be configured to calculate the distance (e.g., Euclidean distance) between each conversion model 3D point in the converted model 3D point cloud data of an instance and each target 3D point in the target 3D point cloud data 132. The processor 122 may be configured to compare the value corresponding to each distance with a threshold. If the value is less than the threshold, the processor 122 may be configured to increment an overlap coefficient corresponding to a 6D pose hypothesis for generating the conversion model 3D point. The overlap coefficient may correspond to the number of conversion model 3D points associated with an instance of the conversion model 3D point cloud data, wherein the calculated distance value between the conversion model 3D point and the target 3D point is less than the threshold. Therefore, the overlap coefficient may be a measure of the degree of match between a 6D pose hypothesis in generating a specific instance of the conversion model 3D point cloud data and the target 3D point cloud data 132. In more detail, a larger overlap coefficient indicates that there is a better match between the conversion model 3D point cloud data of an instance generated by a specific 6D pose hypothesis and the target 3D point cloud data 132 than a smaller overlap coefficient.
[0035] In one embodiment, after applying each six-dimensional pose hypothesis to the model three-dimensional point cloud data 134 to generate the transformed model three-dimensional point cloud data, the processor 122 may be configured to compare all overlap coefficients associated with each six-dimensional pose hypothesis in the first set of six-dimensional pose hypotheses and the second set of six-dimensional pose hypotheses to determine the six-dimensional pose hypothesis corresponding to the maximum overlap coefficient. The processor 122 may be configured to select the six-dimensional pose hypothesis corresponding to the maximum overlap coefficient, and the selected six-dimensional pose hypothesis may include candidate six-dimensional pose hypothesis data.
[0036] In addition, the processor 122 may be configured to generate control data 138 based at least in part on the candidate six-dimensional pose hypothesis data. The control data 138 may include instructions based at least in part on the candidate six-dimensional pose hypothesis data. The instructions may be configured to Figure 1A The robot arm 146 of the robot 112 moves to a specific position and grasps the Figure 1A The processor 122 may be configured to Figure 1B The network interface 128 shown, and thereby through Figure 1A The network 120 will Figure 1B The control data 138 , including candidate six-dimensional pose hypothesis data, is transmitted to the robot 112 .
[0037] Figure 1BThe input / output device 126 may include a mouse, a keyboard, a display device, a camera, other input / output devices, or a combination thereof. In one embodiment, the input / output device 126 may be configured to display a graphical user interface (GUI) generated by the processor 122. The input / output device 126 may be configured to receive user input, possibly through a GUI, such as identifying Figure 1A Alternatively or additionally, the input / output device 126 may be configured to receive instructions, possibly via a GUI, to Figure 1A A three-dimensional representation of object 130 is identified from among plurality of objects 104 .
[0038] The network interface 128 may be configured to allow the computing device 110 to communicate with Figure 1A Robot 112 and Figure 1A For example, the processor 122 may be configured to allow the network interface 128 to communicate with the imaging device 114 via Figure 1A Network 120 to Figure 1A The robot 112 transmits control data 138, including candidate six-dimensional pose hypothesis data. In addition, the network interface 128 can be configured to transmit control data 138 to the robot 112 via Figure 1A The network 120 receives the Figure 1A In one embodiment, the user can Figure 1A The user device 148 provides instructions to identify and select Figure 1A 102, or from Figure 1A Identify and select from among the plurality of objects 104 those geometrically identical Figure 1B The three-dimensional representation of the object 130 is similar to an object. Figure 1A The instructions sent by the user device 148 can be sent by Figure 1A The network 120 in Figure 1B Received on network interface 128.
[0039] like Figure 1AAs shown, the robot 112 may include a processor 140, a memory 142, a network interface 144, and a robotic arm 146. The network interface 144 is configured to receive data, such as control data 138, which includes candidate six-dimensional pose hypothesis data. The robot 112 also includes a processor 140 configured to execute instructions, which include instructions encoded in the control data 138. The processor 140 is also configured to provide control instructions to the robotic arm 146 to move the robotic arm 146 to a specific position and a specific direction based on the control data 138 including the candidate six-dimensional pose hypothesis data. For example, the control data 138 may cause the robotic arm 146 to move to a specific position and a specific direction to grasp a specific object 102 based at least in part on the candidate six-dimensional pose hypothesis data included in the control data 138. In a specific embodiment, Figure 1B The computing device 110 may be integrated into the robot 112 .
[0040] Figure 2A-1 , 2B-1 , 2A-2 and 2B-2 provide an overview of an exemplary method 200 of the inventive concept for selecting a candidate six-dimensional pose hypothesis from a plurality of six-dimensional pose hypotheses. As explained below, the method 200 can use images of multiple objects (including the object to be identified) and three-dimensional representations of the object to be identified to quickly select a candidate six-dimensional pose hypothesis from a plurality of six-dimensional pose hypotheses. The candidate six-dimensional pose hypothesis can correspond to the position and orientation of the three-dimensional representation of the object, which can substantially match the position and orientation of the object in the image. In this way, the object can be identified in images of multiple objects (including the object).
[0041] exist Figure 2A-1 At block 202, the following may be generated: Figure 2B-1 2A. The three-dimensional image 202A may be generated using a time-of-flight imaging system. Alternatively, the three-dimensional image 202A may be generated using a stereo positioning camera. Other known methods may also be used to generate the three-dimensional image 202A.
[0042] exist Figure 2A-1 At block 204, the three-dimensional images 202A of the plurality of objects may be converted into target three-dimensional point cloud data 204A, such as Figure 2B-1As shown. In one example, data corresponding to each spatial coordinate (i.e., the x-coordinate, y-coordinate, and z-coordinate of the Cartesian coordinate system of the imaging device used to generate the three-dimensional image 202A) can be extracted from the three-dimensional image 202A. Further, the extracted data may include direction information about each point relative to the imaging device (which is used to generate the three-dimensional image 202A). For example, a normal vector pointing in the direction of the imaging device can be assigned to a point relatively close to the imaging device (which is used to generate the three-dimensional image 202A), while a normal vector pointing away from the imaging device can be assigned to a point relatively far from the imaging device (which is used to generate the three-dimensional image 202A). The extracted data may include the following: Figure 2B-1 The target three-dimensional point cloud data 204A shown. The target three-dimensional point cloud data 204A can be stored in a computer memory, such as Figure 1B in the memory 124.
[0043] exist Figure 2A-1 At block 206, a three-dimensional representation of a particular object 206A may be selected, such as Figure 2B-1 The three-dimensional representation of the specific object 206A may be a computer-aided design (CAD) representation of the specific object 206A. Other known three-dimensional object representation methods may be used to generate the three-dimensional representation of the specific object 206A. In one embodiment, the computer memory, such as Figure 1B The memory 124 may store three-dimensional representations of a plurality of objects that may be encountered by the apparatus performing method 200. For example, if the apparatus performing method 200 is located in a factory producing windows, the apparatus may include in its memory three-dimensional representations of large planes representing windows.
[0044] exist Figure 2A-1 At block 208, the three-dimensional representation of the particular object 206A may be converted into model three-dimensional point cloud data 208A, such as Figure 2B-1 As with the target 3D point cloud data 204A, the model 3D point cloud data 208A may store information about the position (relative to the imaging device) of each pixel corresponding to the 3D representation of the particular object 206A, and information about the orientation (relative to the imaging device) of each pixel corresponding to the 3D representation of the particular object. In one embodiment, the model 3D point cloud data 208A may be stored in a computer memory, such as Figure 1B of memory 124.
[0045] exist Figure 2A-2 At optional block 210 of the present invention, the target three-dimensional point cloud data 204A may be filtered to remove data of objects that are geometrically different from the three-dimensional representation of the specific object 206A to generate filtered three-dimensional point cloud data 210A, such as Figure 2B-2 For example, Figure 2B-1The three-dimensional image 202A includes images of multiple objects. However, it may be desirable to identify an image of a specific object in the three-dimensional image 202A of multiple objects. The specific object image may have Figure 2B-1 The three-dimensional representation of the three-dimensional image of the particular object 206A has a substantially similar geometric shape. Figure 2A-2 The output of the optional filtering block 210 may be an image of a specific object having a similar geometry to the three-dimensional representation of the specific object 206A. The output of the optional filtering block 210 may be referred to as filtered three-dimensional point cloud data 210A, such as Figure 2B-2 shown.
[0046] To achieve Figure 2A-2 The optional filtering block 210 may use a deep learning algorithm known in the art. For example, Figure 2B-1 The target three-dimensional point cloud data 204A is provided as input to the trained mask convolutional neural network to obtain Figure 2B-1 The target three-dimensional point cloud data 204A is identified as being geometrically identical to Figure 2B-1 As output, the masked convolutional neural network may generate an image of an object similar to the three-dimensional representation of the specific object 206A. Figure 2B-2 The filtered three-dimensional point cloud data 210A is Figure 2B-1 The three-dimensional representation of the specific object 206A is geometrically similar. In some embodiments, Figure 2B-2 The filtered three-dimensional point cloud data 210A may be referred to as a segmented image.
[0047] exist Figure 2A-2 At the optional box 212, Figure 2B-1 3D point cloud data 208A or Figure 2B-2 The filtered three-dimensional point cloud data 210A is partitioned. A partition number can be assigned to each partition. Therefore, Figure 2A-2 The optional block 212 may generate partitioned target three-dimensional point cloud data 212A, such as Figure 2B-2 shown. Figure 2B-2 The partitioned target 3D point cloud data 212A can be used to determine the degree of overlap between the transformed model 3D points and the target 3D points of an instance of the transformed model 3D point cloud data, which will be Figure 3 Explained in.
[0048] exist Figure 2A-2 At block 214, multiple six-dimensional pose hypotheses 214 may be generated, such as Figure 2B-2 In one example, a local 3D feature matching method can be used, such as Fig. 6A and Figure 6BThe method described in , generates multiple six-dimensional pose hypotheses 214A. In other embodiments, other methods known in the art can be used to generate multiple six-dimensional pose hypotheses 214A. For example, a random sampling consensus method can be used to generate multiple six-dimensional pose hypotheses 214A. Regardless of the method used to generate multiple six-dimensional pose hypotheses 214A, in implementation, at least hundreds of thousands or even millions of six-dimensional pose hypotheses can be generated.
[0049] exist Figure 2A-2 At block 216 of Figure 2B-2 A candidate six-dimensional pose hypothesis 216A is selected from the plurality of six-dimensional pose hypotheses 214A. The candidate six-dimensional pose hypothesis 216A corresponds to a translation matrix, which is configured to Figure 2B-1 The model three-dimensional point cloud data 208A is translated and corresponds to a rotation matrix, which is configured to rotate the model three-dimensional point cloud data 208A so that the translated and rotated model three-dimensional point cloud data 208A is consistent with Figure 2B-1 The target three-dimensional point cloud data 204A, Figure 2B-2 The filtered target three-dimensional point cloud data 210A, or Figure 2B-2 The target three-dimensional point cloud data 212A of the partitions substantially overlap. In an embodiment, the phrase "substantially overlaps" means that Figure 2B-1 Most of the translation and rotation of the model 3D point cloud data 208A and Figure 2B-1 The target three-dimensional point cloud data 204A, Figure 2B-2 The filtered target three-dimensional point cloud data 210A or Figure 2B-2 The Euclidean distance calculated between the partitioned target three-dimensional point cloud data 212A is less than a threshold distance value. The threshold distance value can be set by a user of the device performing method 200. In other embodiments, the phrase "substantially overlaps" refers to Figure 2B-1 Most of the translated and rotated model 3D point cloud data 208A and Figure 2B-1 The target three-dimensional point cloud data 204A, Figure 2B-2 The filtered target three-dimensional point cloud data 210A or Figure 2B-2 The partitioned target three-dimensional point cloud data 212A have substantially similar positions and directions.
[0050] Figure 3Specific details of an exemplary method 300 for selecting a candidate six-dimensional pose hypothesis from multiple six-dimensional pose hypotheses according to the concepts of the present invention are described. As explained below, method 300 can improve computational efficiency by performing parallelized mathematical operations on model three-dimensional point cloud data to generate multiple instances of transformed model three-dimensional point cloud data. Further operations can be performed on the transformed model three-dimensional point cloud data of each instance to generate overlap metrics between the transformed model three-dimensional point cloud data instance and the target three-dimensional point cloud data (which corresponds to the image of the object to be identified). These overlap metrics can indicate how consistent the six-dimensional pose hypothesis is with the three-dimensional representation of the object to be identified and the image of the object to be identified. Therefore, the six-dimensional pose hypothesis with the largest overlap metric can be selected from multiple six-dimensional pose hypotheses as a candidate six-dimensional pose hypothesis.
[0051] At block 302, a plurality of six-dimensional pose hypotheses may be generated. The number of six-dimensional pose hypotheses may correspond to the three-dimensional representation of the object (e.g., Figure 1B As another example, the number of six-dimensional pose hypotheses can be set by a user, such as a user of the system 100. The number of six-dimensional pose hypotheses can be generated by techniques known in the art, such as by a local three-dimensional point pair feature algorithm, an illustrative example of which will be referred to in detail in FIG. Fig. 6A and 6B explain.
[0052] At block 304, the model three-dimensional point cloud data is instantiated. In one embodiment, a threshold number of copies of the model three-dimensional point cloud data may be generated. The threshold number may correspond to the size of a computer memory, such as the memory size of a computing device executing method 300. As another embodiment, the threshold number may be equal to Figure 1B The size of the memory 124 of the computing device 110 shown is shown. In other embodiments, the threshold number may be smaller than the size of the computing device or the computing device executing the method 300. Figure 1B In another embodiment, the model 3D point cloud data of a single instance is generated.
[0053] At box 306, the model three-dimensional point cloud data can be stored in a memory, such as a memory of a computing device. For example, a threshold number of copies of the model three-dimensional point cloud data can be stored in the memory. The model three-dimensional point cloud data can be stored in a data structure corresponding to a one-dimensional array. For example, in one embodiment, the data corresponding to each coordinate of each model three-dimensional point of each copy of the model three-dimensional point cloud data can be stored in an address of the memory. Although a one-dimensional array is an exemplary data structure for storing model three-dimensional point cloud data, other data structures can also be used. For example, a C++ vector data structure can be used. In another embodiment, a single instance of model three-dimensional point cloud data can be generated. For example, in an object-oriented computer language, such as C++ or Java, a model three-dimensional point cloud data class can be instantiated, which can include a first variable configured to store a value corresponding to the x coordinate of a model three-dimensional point, a second variable configured to store a value corresponding to the y coordinate of a model three-dimensional point, and a third variable configured to store a value corresponding to the z coordinate of a model three-dimensional point.
[0054] At block 308, a set of six-dimensional pose hypotheses may be selected from the plurality of six-dimensional pose hypotheses, the set of six-dimensional pose hypotheses corresponding to a number of six-dimensional pose hypotheses. In one embodiment, the number of six-dimensional pose hypotheses included in the set of six-dimensional pose hypotheses may correspond to the number of six-dimensional pose hypotheses stored in the memory (e.g., Figure 1B The first set of six-dimensional pose hypotheses can be randomly selected from multiple six-dimensional pose hypotheses. Alternatively, when multiple six-dimensional pose hypotheses are generated, a value can be assigned to each of the multiple six-dimensional pose hypotheses. The assigned value can correspond to an initial indicator of the probability that the six-dimensional pose hypothesis may be well matched with the target three-dimensional point cloud data. For example, when generating multiple six-dimensional pose hypotheses, some model three-dimensional point cloud data generated by a specific six-dimensional pose hypothesis and corresponding to the conversion of the model three-dimensional point cloud data of an instance can be randomly selected. A distance (such as a Euclidean distance) can be calculated between each randomly selected conversion model three-dimensional point and each target three-dimensional point. If the specified number of distances are all less than a threshold, a value can be assigned to the specific six-dimensional pose hypothesis. The value can indicate that there is a high probability of matching between the conversion model three-dimensional point generated by the specific six-dimensional pose hypothesis and the target three-dimensional point data. In one embodiment, the number of six-dimensional pose hypotheses (corresponding to the first group of six-dimensional pose hypotheses) selected from the multiple six-dimensional pose hypotheses may be equal to a threshold number of copies of the three-dimensional model point cloud data.
[0055] At box 310, the model 3D point cloud data can be converted into a transformed model 3D point cloud data instance using the first set of six-dimensional pose assumptions. For example, the model 3D point cloud data can be converted into a transformed model 3D point cloud data instance by parallel mathematical operations, wherein each of the selected pose assumptions (i.e., the pose assumptions selected at box 308, including the first set of pose assumptions) is applied to the model 3D point cloud data. For illustration, the coordinates of the model 3D points corresponding to the model 3D point cloud data (e.g., an instantiation of the model 3D point cloud data, or, in another embodiment, each copy of the model 3D point cloud data) can be calculated by (e.g., multiplying by) the translation and rotation matrices corresponding to each six-dimensional pose assumption. In this way, multiple instances of transformed model 3D point cloud data can be generated in parallel, each instance of the transformed model 3D point cloud data having a transformed model 3D point that is translated and oriented in a specific manner corresponding to a specific translation and rotation matrix associated with the corresponding six-dimensional pose assumption.
[0056] In a specific embodiment, and as an illustrative example, three copies of the model three-dimensional point cloud data can be instantiated. In this example, each model three-dimensional point of the first copy of the model three-dimensional point cloud data can be multiplied by the translation matrix and rotation matrix associated with the first six-dimensional pose hypothesis in the selected six-dimensional pose hypothesis (i.e., the first group of six-dimensional pose hypotheses) to generate a translated and rotated three-dimensional point, including the first instance of the converted model three-dimensional point cloud data. In parallel, each model three-dimensional point of the second copy of the model three-dimensional point cloud data is multiplied by the translation matrix and rotation matrix associated with the second six-dimensional pose hypothesis of the selected six-dimensional pose hypothesis (i.e., the first group of pose hypotheses) to generate a translated and rotated three-dimensional point including the second instance of the converted model three-dimensional point cloud data. Further, and in parallel, each model three-dimensional point in the third copy of the model three-dimensional point cloud data is multiplied by the translation matrix and rotation matrix associated with the third pose hypothesis in the selected pose hypothesis (i.e., the first group of pose hypotheses) to generate a translated and rotated three-dimensional point including the third instance of the converted model three-dimensional point cloud data. The foregoing examples are for illustrative purposes only. In many embodiments, hundreds, thousands, hundreds of thousands, or millions of copies of the model 3D point cloud data may be generated, and the above operations may be performed in parallel on each copy. As another example and to further illustrate, converting each model 3D point in the model 3D point cloud data into a transformed model 3D point, and converting each copy in the model 3D point cloud data into an instance of the transformed model 3D point cloud data may be performed in parallel.
[0057] Optionally, in another embodiment corresponding to another different embodiment, the transformed model three-dimensional point cloud data of multiple instances are generated by iteratively applying each of the multiple six-dimensional pose hypotheses to the model three-dimensional point cloud data of a single instance. For illustration, the coordinates of the model three-dimensional point cloud data corresponding to one instance can be iteratively multiplied by the translation and rotation matrices of each six-dimensional pose hypothesis corresponding to the multiple six-dimensional pose hypotheses.
[0058] At block 312, a degree of overlap between a conversion model 3D point of the conversion model 3D point cloud data instance and a target 3D point of the target 3D point cloud data may be determined. In one embodiment, the degree of overlap may correspond to a Euclidean distance between a conversion model 3D point of the conversion model 3D point cloud data instance and a target 3D point of the target 3D point cloud data. The distance calculation may be performed in real time between each conversion model 3D point of the conversion model 3D point cloud data instance and a target 3D point of the target 3D point cloud data.
[0059] In one embodiment, the target three-dimensional point cloud data has been partitioned, such as Figure 2A-2 and 2B-2 Describing, especially Figure 2A-2 In block 212 of the embodiment, the Euclidean distance between each target 3D point corresponding to and located in a specific partition number and the transformed model 3D point of each instance transformed model 3D point cloud data can be determined. In more detail, the partition number is determined according to the position of the transformed model 3D point, and the distance calculation can be performed between the transformed model 3D point and the target 3D point in the partition corresponding to the determined partition number. Figure 2A-2 and 2B-2 , as an example, can be found in Figure 2B-2 The distance is determined between the target 3D point corresponding to the target 3D point cloud data 212A of the partition 2 depicted in FIG. 1 and the transformed model 3D point of each instance transformed model 3D point cloud data. In other embodiments, the distance can be determined as follows: Figure 2B-2 The Euclidean distance is determined between the target 3D point corresponding to the target 3D point cloud data 212A of other partition numbers and the conversion model 3D point of each instance conversion model 3D point cloud data. Determining the Euclidean distance between the target 3D point of the target 3D point cloud data 212A corresponding to a specific partition number and the conversion model 3D point of an instance conversion model 3D point cloud data reduces the number of times the distance calculation is performed and improves the calculation efficiency. In this way, the speed of executing the exemplary method 300 can be increased, and the time required to identify an object from multiple objects can be reduced.
[0060] At block 314, the distance generated at block 312 may be compared to a threshold. The threshold may be set by the user, such as Figure 1A and 1BThe comparison of block 314 may be performed for each distance calculated between each copy of each transformed model 3D point of the transformed model 3D point cloud data and each target 3D point of the target 3D point cloud data. Alternatively, the comparison of block 314 may be performed for each distance calculated between each transformed model 3D point and each target 3D point associated with a partition number.
[0061] At box 316, if the calculated distance between the conversion model 3D point generated by one of the first plurality of 6D pose hypotheses and a target 3D point is less than a threshold, the overlap coefficient corresponding to the 6D pose hypothesis can be incremented. The overlap coefficient can correspond to a measure of the number of conversion model 3D points (which coincide with the target 3D point) generated by a particular 6D pose hypothesis. For example, if the first overlap coefficient associated with the first 6D pose hypothesis is greater than the second overlap coefficient associated with the second 6D pose hypothesis, a greater number of conversion model 3D points are generated by the first 6D pose hypothesis than by the second 6D pose hypothesis, and the distance (such as the Euclidean distance) between the target 3D point is less than the threshold. Therefore, in this embodiment, the first 6D pose hypothesis generates more conversion model 3D points that coincide with the target 3D point than the second 6D pose hypothesis. Therefore, in this embodiment, the transformation model three-dimensional point cloud data instance generated by the translation matrix and rotation matrix corresponding to the first six-dimensional pose hypothesis is more consistent with the target three-dimensional point cloud data than the transformation model three-dimensional point cloud data instance generated by the translation matrix and rotation matrix corresponding to the second six-dimensional pose hypothesis.
[0062] In one embodiment, each overlap coefficient corresponding to each of the plurality of six-dimensional pose hypotheses may be stored at an address of a memory of a computing device executing method 300. For example, after performing all of the distance calculations described herein, each overlap coefficient may be overwritten to an address of a memory where data corresponding to the coordinates of a transformed model three-dimensional point was previously stored.
[0063] At block 318, the magnitudes of the overlap coefficients may be compared. For example, the magnitude of each overlap coefficient may be compared to the magnitude of each other overlap coefficient. A six-dimensional pose hypothesis in the first set of six-dimensional pose hypotheses corresponding to the maximum overlap coefficient may be selected as a potential candidate six-dimensional pose hypothesis. The potential candidate six-dimensional pose hypothesis may be the six-dimensional pose hypothesis in the first set of six-dimensional pose hypotheses that produces the largest number of transformed model three-dimensional points that overlap with the target three-dimensional point.
[0064] At box 320, it can be determined whether the model three-dimensional point cloud data has been converted into an instance of model three-dimensional point cloud data by each six-dimensional pose hypothesis of multiple six-dimensional pose hypotheses. If it is determined that the six-dimensional pose hypothesis among the multiple six-dimensional pose hypotheses has not been used to convert the model three-dimensional point cloud data, the method 300 can return to box 308. At box 308, a pose hypothesis that was not previously selected among the multiple pose hypotheses can be selected, and the selected second group of six-dimensional pose hypotheses can be applied at boxes 310-316. At box 318, the size of the overlap coefficient corresponding to each six-dimensional pose hypothesis in the selected second group of six-dimensional pose hypotheses can be compared. The six-dimensional pose hypothesis corresponding to the maximum overlap coefficient can be selected from the selected second group of six-dimensional pose hypotheses as a potential candidate six-dimensional pose hypothesis.
[0065] At box 320, if it is determined that the model three-dimensional point cloud data has been converted into a converted model three-dimensional point cloud data instance by each of the multiple six-dimensional pose hypotheses generated at box 302, the method 300 can proceed to box 322. At box 322, the size of each overlap coefficient associated with each potential candidate six-dimensional pose hypothesis can be compared with the size of other overlap coefficients associated with each other potential candidate six-dimensional pose hypothesis. At box 324, a candidate six-dimensional pose hypothesis can be selected from the potential candidate six-dimensional pose hypotheses. The potential candidate six-dimensional pose hypothesis with the largest overlap coefficient can be selected as the candidate six-dimensional pose hypothesis. In an example where there is only one potential candidate six-dimensional pose hypothesis, the potential candidate six-dimensional pose hypothesis is selected as the candidate six-dimensional pose hypothesis.
[0066] At optional block 326, candidate six-dimensional pose hypothesis data corresponding to the candidate six-dimensional pose hypothesis may be sent to the robot, such as Figure 1A The candidate six-dimensional pose hypothesis data can be combined with the control instructions (such as Figure 1B In this manner, the candidate six-dimensional pose hypothesis data corresponding to the candidate six-dimensional pose hypothesis can be used to perform an action, such as by the robot (e.g., Figure 1A The robot 112) identifies and manipulates an object from multiple objects.
[0067] By performing parallel mathematical operations on each model 3D point in the model 3D point cloud data and each copy of the model 3D point cloud data using a six-dimensional pose hypothesis selected from multiple six-dimensional pose hypotheses to generate multiple instances of transformed model 3D point cloud data, computer processing time can be reduced because the mathematical operations are performed in parallel rather than serially. For example, a conventional processing process can use the six-dimensional pose hypothesis to serially generate instances of transformed model 3D point cloud data. Such a conventional processing process will be relatively slow.
[0068] As an example, the method 300 was executed on a computing device to identify a specific object from a plurality of objects. The method 300 successfully identified the object from the plurality of objects in approximately 0.3 seconds. A conventional known Drost point pair feature algorithm was also executed on the same computing device to identify the same specific object from a plurality of objects. The Drost point pair feature algorithm took approximately 0.5 seconds to identify the same object from a plurality of objects. Thus, the method 300 of the present invention improves object recognition time compared to conventional methods.
[0069] Further, in response to determining that one or more of the plurality of six-dimensional pose hypotheses are not used to generate the model three-dimensional point cloud data, iteratively generating additional instances of the converted model three-dimensional point cloud data can improve computational efficiency by reducing the size of the memory on which the method 300 is executed. For illustration, as explained above, in a particular embodiment, the converted model three-dimensional point cloud data is generated by multiple copies of the model three-dimensional point cloud data (which corresponds to the size of the memory of the device on which the method 300 is executed). However, in most practical applications, the number of the plurality of six-dimensional pose hypotheses is expected to greatly exceed the size of the memory of the device on which the method 300 is executed. Therefore, in order to generate an instance of the converted model three-dimensional point cloud data from each of the plurality of six-dimensional pose hypotheses without resorting to the use of additional memory or without expanding the size of the memory of the device on which the method 300 is executed, the iterative process described in the context of at least block 320 may be advantageous. In this manner, by iteratively performing blocks 308 - 320 until each of the plurality of 6D pose hypotheses is applied to the model 3D point cloud data, memory may be saved because no additional copies of the model 3D point cloud data need to be instantiated.
[0070] Figure 4A and 4B An exemplary data structure 400 is depicted in accordance with at least one embodiment of the method 300 of the present inventive concepts. Figure 4A The data structure 400 can be configured to perform multiple mathematical operations in parallel to generate the transformation model three-dimensional point cloud data, from which a candidate six-dimensional pose hypothesis can be selected from multiple six-dimensional pose hypotheses. Figure 4A The data structure 400 can reduce the consumption of computing time to effectively select the correct candidate six-dimensional pose hypothesis.
[0071] The data structure 400 may be stored in a memory of a computing device executing the method 400. For example, the data structure 400 may be stored in a memory of a computing device executing the method 400. Figure 1B The data structure 400 may include the following: Figure 4A The data structure 400 may also include the first copy of the model three-dimensional point cloud data 404. Figure 4A A second copy of the model three-dimensional point cloud data 406 is shown. In addition, the data structure 400 may include Figure 4A The nth copy of the model 3D point cloud data 408 is shown. The first copy of the model 3D point cloud data 404, the second copy of the model 3D point cloud data 406, and the nth copy of the model 3D point cloud data 408 may be cumulatively equal to a threshold number of copies of the model 3D point cloud data. The threshold number of copies of the model 3D point cloud data may be equal to the size of the memory storing the data structure 400. Optionally, the threshold number of copies of the model 3D point cloud data may be less than the size of the memory storing the data structure 400. In one embodiment, as Figure 4A As shown, a first copy of the model 3D point cloud data 404, a second copy of the model 3D point cloud data 406, and a third copy of the model 3D point cloud data 408 can be stored in a first group of memory addresses 402, also referred to as a first container, such as Figure 4A shown.
[0072] In one embodiment, data corresponding to a first coordinate (such as an X coordinate of a Cartesian coordinate system) of a point (such as a first point) of a copy of the model 3D point cloud data (such as a first copy of the model 3D point cloud data 404) may be stored in Figure 4A The data corresponding to the second coordinate (such as the Y coordinate of the Cartesian coordinate system) of the point (such as the first point) of the first copy of the model three-dimensional point cloud data 404 can be stored in the memory as shown in FIG. Figure 4A Similarly, data corresponding to a third coordinate (such as a Z coordinate of a Cartesian coordinate system) of a point (such as a first point) of the first copy of the model three-dimensional point cloud data 404 may be stored at a second address 412 of the memory. Figure 4A In a similar manner, data corresponding to each coordinate of each point of each copy of the model three-dimensional point cloud data can be stored in Figure 4A At a specific address in the memory depicted in .
[0073] In one embodiment, the number of multiple six-dimensional pose hypotheses (i.e., the multiple six-dimensional pose hypotheses generated in box 302 of method 300) may be greater than the threshold number of copies of the model three-dimensional point cloud data. Therefore, a six-dimensional pose hypothesis can be selected from the multiple six-dimensional pose hypotheses, wherein the number of selected six-dimensional pose hypotheses may not be greater than the threshold number of copies of the model three-dimensional point cloud data. For example, the number of the multiple six-dimensional pose hypotheses may be twice the threshold number of copies of the model three-dimensional point cloud data (such as n copies). In such an embodiment, n six-dimensional pose hypotheses can be selected from the 2n six-dimensional pose hypotheses depicted in Figure 4. The selected pose hypothesis may be referred to as a first group of multiple six-dimensional pose hypotheses. Figure 4A As shown, by using the selected six-dimensional pose assumptions 416, 418 and 420, a mathematical operation is performed on each of the threshold number of copies of the model three-dimensional point cloud data to generate a Figure 4A Multiple instances 424 , 426 , and 428 of the converted model three-dimensional point cloud data 422 are shown.
[0074] To illustrate, Figure 4A As shown, a mathematical operation may be performed on the first copy of the model 3D point cloud data 404 through the first 6-dimensional pose hypothesis 416 to generate the transformed model 3D point cloud data 424. The mathematical operation may include: multiplying each coordinate of each point constituting the first copy of the model 3D point cloud data 404 by a translation matrix corresponding to the first 6-dimensional pose hypothesis 416 and a rotation matrix corresponding to the first 6-dimensional pose hypothesis 416. By multiplying each coordinate of each point of the first copy of the model 3D point cloud data 404, each coordinate of each point of the first copy of the model 3D point cloud data 404 may be converted into the transformed model 3D point cloud data 424 of the first instance.
[0075] In a similar way, Figure 4A As shown, when the first copy of the model 3D point cloud data 404 is mathematically operated by the first six-dimensional pose hypothesis 416 to generate the converted model 3D point cloud data 424 of the first instance, the second copy of the model 3D point cloud data 406 can be mathematically operated by the second six-dimensional pose hypothesis 418 to generate the converted model 3D point cloud data 426 of the second instance. Similarly, each remaining copy of the model 3D point cloud data (i.e., the remaining n-2 copies) can be operated in parallel by the selected multiple remaining n-2 six-dimensional pose hypotheses to generate the converted model 3D point cloud data of the remaining n-2 instances. Therefore, as Figure 4AAs shown, m×n×3 rows of conversion model three-dimensional point cloud data can be generated. The m×n×3 rows of conversion model three-dimensional point cloud data may include conversion model three-dimensional point cloud data 422. The first instance conversion model three-dimensional point cloud data 424 may be different from the second instance conversion model three-dimensional point cloud data 426 and the nth instance conversion model three-dimensional point cloud data 428, wherein the individual points constituting the conversion model three-dimensional point cloud data 422 have been translated and rotated through the selected six-dimensional pose hypothesis (e.g., the first six-dimensional pose hypothesis 416, the second six-dimensional pose hypothesis 418, and the nth six-dimensional pose hypothesis 420). In one example, the converted model three-dimensional point cloud data 422 may be stored in a second group of storage addresses 442, also referred to as a second container, such as Figure 4A shown.
[0076] Each transformed model 3D point of the transformed model 3D point cloud data 422 of each instance is Figure 4A The distance (such as the Euclidean distance) can be determined between each target 3D point of the target 3D point cloud data 434, for example Figure 4A The distances 430 and 432 are used to generate first distance data 436, second distance data 438 and nth distance data 440, as shown in FIG. Figure 4B As shown, these data can be stored in a third group of memory addresses 446, also referred to as a third container. For illustration, the distance, e.g. Figure 4A The distances 430 and 432 can be Figure 4A Each conversion model 3D point and Figure 4A The target three-dimensional point cloud data 434 shown in FIG. 4 is determined between each target three-dimensional point, generating Figure 4B Similarly, the distance between each conversion model 3D point of the conversion model 3D point cloud data 426 of the second instance and each target 3D point of the target 3D point cloud data 434 can be determined to generate Figure 4B In a similar manner, the distance between each conversion model 3D point of the conversion model 3D point cloud data 428 and each target 3D point of the target 3D point cloud data 434 can be determined to generate Figure 4B The nth distance data 440 is shown. Cumulatively, Figure 4B The first distance data 436 , the second distance data 438 , and the nth distance data 440 may be referred to as a first set of distance data.
[0077] In one embodiment, the first set of distance data may be stored in a third set of storage addresses 446, which is referred to as a third container. Figure 4BIn some embodiments, each distance corresponding to each coordinate may be stored in a specific address of the third group of storage addresses 446. For example, the distance between the x-coordinate of the first point of the converted model three-dimensional point cloud data 424 of the first instance and the corresponding x-coordinate of a point of the target three-dimensional point cloud data 434 may be stored in the first address 460 of the third group of storage addresses 446, as shown in FIG. Figure 4B Similarly, the distance between the y coordinate of the first point of the converted model three-dimensional point cloud data 424 of the first instance and the corresponding y coordinate of the point of the target three-dimensional point cloud data 434 can be stored in the second address 462 of the third group of storage addresses 446, as shown in FIG. Figure 4B In a similar manner, the distance between the z coordinate of the first point of the converted model three-dimensional point cloud data 424 of the first instance and the corresponding z coordinate of the point of the target three-dimensional point cloud data 434 can be stored in the third address 464 of the third group of storage addresses 446, as shown in FIG. Figure 4B shown.
[0078] In another embodiment, only Figure 4A The distance calculation is performed between the target 3D point associated with the specific partition number 444 and the converted model 3D point corresponding to the target 3D point associated with the specific partition number 444 (i.e., the target 3D point located in the partition associated with the partition number). Therefore, the number of distance calculations can be reduced, and the computational efficiency of the method 300 can be improved. In this way, and Figure 4A Compared with the case where distance calculation is performed for each target three-dimensional point in the target three-dimensional point cloud data 434, a candidate six-dimensional pose hypothesis can be generated in a shorter time.
[0079] The magnitude of each distance included in the distance data can be compared with a threshold. The threshold can be determined by a computing device (such as Figure 1B The threshold value may also be stored in a memory of a computing device executing method 300. If the magnitude of the distance including the distance data is less than the threshold value, then the overlap coefficient corresponding to a six-dimensional pose hypothesis that generates a corresponding transformed model three-dimensional point of an instance transformed model three-dimensional point cloud data may be incremented. For illustration, if Figure 4B The first distance of the first distance data 436 is less than a threshold value, then Figure 4B The first overlap factor 450 may be incremented. Figure 4B The first overlap factor 450 may correspond to Figure 4A The first six-dimensional pose of is assumed 416. In a similar manner, if Figure 4B The second distance of the second distance data 438 is less than the threshold, then Figure 4B The second overlap coefficient 452 (corresponding to Figure 4A The second six-dimensional pose hypothesis 418) can be incremented. Similarly, if Figure 4B If the third distance of the nth distance data 440 is less than the threshold, then Figure 4B The nth overlap coefficient 454 (corresponding to Figure 4A In one embodiment, the overlap coefficients, such as the first overlap coefficient 450, the second overlap coefficient 452, and the nth overlap coefficient 454, can be stored in a fourth group of memory addresses 448, referred to as a fourth container, such as Figure 4B shown.
[0080] As will be appreciated by those skilled in the art, the overlap coefficient (e.g. Figure 4B The first overlap coefficient 450, the second overlap coefficient 452, and the nth overlap coefficient 454 shown in FIG. 4 may represent a six-dimensional pose hypothesis, such as Figure 4A The first six-dimensional pose assumption 416, Figure 4A The second six-dimensional pose assumption 418, and Figure 4A The degree to which the transformed model 3D points generated by the nth 6-dimensional pose hypothesis 420 overlap with the target 3D points of the target 3D point data 434. For example, if Figure 4B The magnitude of the nth overlap coefficient 454 is greater than Figure 4B The first overlap factor is 450 or Figure 4B If the size of the second overlap coefficient 452 is greater than that of the first instance transformed model 3D point cloud data 424 or the second instance transformed model 3D point cloud data 426, the distance between the points of the nth instance 3D point cloud data 428 and the target 3D point cloud data 434 is less than the threshold. Therefore, the overlap coefficient can be used to compare the 6D pose hypotheses.
[0081] In one embodiment, the magnitudes of the overlap coefficients can be compared with each other. Figure 4B As shown, the first amplitude of the first overlap coefficient 450, the second amplitude of the second overlap coefficient 452, and the third amplitude of the nth overlap coefficient 454 can be compared with each other. Figure 4B As shown, a six-dimensional pose hypothesis corresponding to the largest overlap coefficient 458 among the overlap coefficients (for example, the first overlap coefficient 450, the second overlap coefficient 452, and the nth overlap coefficient 454) can be selected as a potential candidate six-dimensional pose hypothesis for the first group of six-dimensional pose hypotheses.
[0082] In embodiments of the invention, the number of 6D pose hypotheses may exceed the size of a memory configured to perform exemplary methods such as methods 200 and 300. However, it may be advantageous to evaluate the overlap corresponding to these additional 6D pose hypotheses. Figure 5A and 5BAn embodiment is depicted in which model 3D points may be iteratively operated on through a second set of 6D pose hypotheses selected from a plurality of 6D pose hypotheses to accommodate situations in which the number of 6D pose hypotheses exceeds the memory size of a device configured to perform exemplary methods 200 and 300.
[0083] like Figure 5A The model 3D point cloud data 404 stored in the first memory address 402 (ie, the first container) can be operated by the second set of 6D pose assumptions. For example, the first copy of the model 3D point cloud data 404 can be calculated by Figure 5A The first six-dimensional pose hypothesis in the second set of six-dimensional pose hypotheses 502 depicted in FIG. Figure 5A As shown, the second copy of the three-dimensional point cloud data 406 can be operated in parallel by the second six-dimensional pose hypothesis in the second set of six-dimensional pose hypotheses 504. In a similar manner, as shown in FIG. Figure 5A As shown, the n-th replica model three-dimensional point cloud data 408 can be parallelized by the n-th six-dimensional pose hypothesis of the second set of six-dimensional pose hypotheses 506. Therefore, through these parallel operations, the following can be generated: Figure 5A The second set of transformed model 3D point cloud data 522 is shown. In one embodiment, the second set of transformed model 3D point cloud data 522 can be stored in Figure 5A For example, the second set of transformed model three-dimensional point cloud data 522 can be processed by a processor (such as Figure 1B processor 122) is overwritten to the second set of memory addresses 442.
[0084] For further explanation, in one embodiment, reference is made to Figure 5A , the first six-dimensional pose hypothesis 502 of the second set of six-dimensional pose hypotheses can be operated on the first copy of the transformed model three-dimensional point cloud data 404 to generate the second set of transformed model three-dimensional point cloud data 508 of the first instance. In parallel, the second six-dimensional pose hypothesis 504 of the second set of six-dimensional pose hypotheses can be operated on the second copy of the model three-dimensional point cloud data 406 to generate the second set of transformed model three-dimensional point cloud data 510 of the second instance. Similarly, in parallel, the nth six-dimensional pose hypothesis 506 of the second set of six-dimensional pose hypotheses can be operated on the nth copy of the model three-dimensional point cloud data 408 to generate the second set of transformed model three-dimensional point cloud data 512 of the nth instance. In this manner, multiple parallel operations occur, thereby reducing the computational time for generating the second set of transformed model three-dimensional point cloud data 522 and increasing the speed of selecting appropriate candidate six-dimensional pose hypotheses.
[0085] As shown in FIG4 , the distance (e.g., distance 514, 516) between the conversion model 3D points of the second group of conversion model 3D point cloud data 522 and the target 3D points of the target 3D point cloud data 434 can be calculated. In one embodiment, a second group of distance data can be generated and stored in a third group of storage addresses 446 (i.e., in a third container). For example, the second group of distance data calculated based on the distance between the conversion model 3D points of the conversion model 3D point cloud data 522 and the target 3D points of the target 3D point cloud data 434 can be overwritten on the third group of storage addresses 446. In a similar manner, a second group of first distance data 518, a second group of second distance data 520, and a second group of nth distance data 524 can be generated and stored. Cumulatively, the second group of first distance data 518, the second group of second distance data 520, and the second group of nth distance data 524 can be referred to as a second group of distance data.
[0086] Each distance including the second set of first distance data 518, the second set of second distance data 520, and the nth distance data 524 may be compared to a threshold value. In a manner similar to that described in FIG. 4 , in response to the distance of the second set of distance data being less than the threshold value, the overlap coefficient is increased. Figure 5A As shown, a first overlapping coefficient 526 of the second group of overlapping coefficients, a second overlapping coefficient 528 of the second group of overlapping coefficients, and an nth overlapping coefficient 530 of the second group of overlapping coefficients can be determined. Cumulatively, the first overlapping coefficient 526 of the second group of overlapping coefficients, the second overlapping coefficient 528 of the second group of overlapping coefficients, and the nth overlapping coefficient 530 of the second group of overlapping coefficients can be referred to as the second group of overlapping coefficients. In one embodiment, the first overlapping coefficient 526 of the second group of overlapping coefficients can correspond to the first six-dimensional pose hypothesis 502 of the second group of six-dimensional pose hypotheses. Among them, the second overlapping coefficient of the second group of overlapping coefficients 528 can correspond to the second six-dimensional pose hypothesis 504 of the second group of six-dimensional pose hypotheses, and the nth overlapping coefficient of the second group of overlapping coefficients 530 can correspond to the nth six-dimensional pose hypothesis 506 of the second group of six-dimensional pose hypotheses. In an example, the second group of overlapping coefficients can be stored in a fourth group of memory addresses 448 (i.e., a fourth container). For example, a processor, such as Figure 1B The processor 122 overwrites the second set of overlap coefficients on the fourth set of memory addresses 448, such as Figure 5A shown.
[0087] like Figure 5B As shown, the sizes of the second set of overlapping coefficients can be compared, and the maximum overlapping coefficient of the second set of overlapping coefficients can be determined. Figure 5B , the six-dimensional pose hypothesis 534 corresponding to the maximum overlap coefficient 534 in the second set of overlap coefficients can be selected as a potential candidate six-dimensional pose hypothesis for the second set of six-dimensional pose hypotheses. Figure 5BAs shown in 536 in FIG. 5 , the maximum overlap coefficient 458 and the maximum overlap coefficient 534 of the second set of overlap coefficients can be compared. The potential six-dimensional pose hypothesis corresponding to the maximum overlap coefficient (i.e., the maximum overlap coefficient between the maximum overlap coefficient 458 and the maximum overlap coefficient 534 of the second set of overlap coefficients) can be selected as the candidate six-dimensional pose hypothesis, as shown in FIG. Figure 5B shown.
[0088] Fig. 6A and 6B A method 600 for generating a plurality of six-dimensional pose hypotheses in a particular embodiment of the present invention is described. As explained below, the method 600 may identify a pair of local points on a surface of a three-dimensional representation of an object to be identified. The method 600 may align the pair of points with a reference point selected from an image of the object. In this manner, the method 600 may be used to generate a plurality of six-dimensional pose hypotheses.
[0089] At block 602, you can select Figure 6B 604A, which are located on a surface 608A of a particular object 3D representation corresponding to the model 3D point cloud data. At block 604, a feature vector corresponding to the model 3D point pair 602A, 604A may be determined. The feature vector may include the Euclidean distance of the line 610A, such as Figure 6B As shown, the line 610A connects the first point m1 and the second point m2 (distance component) of the model three-dimensional point pair such as the point pair 602A. The feature vector may also include a first measure of the first angle 612A, such as Figure 6B As shown, the angle 612A is between the line 610A and the first normal vector n1 associated with the first point m1 (the first angle component). In addition, the feature vector may include a second measure of a second angle 614A between the second normal vector n2 associated with the second point m2 and the line 610A (the second angle component). The last component of the feature vector may be a measure of the angle 616A between the first vector n1 and the second vector n2 (the third angle component), as shown. Figure 6B shown.
[0090] At block 606, a reference target 3D point pair may be selected from the target 3D points of the target 3D point cloud data, whose feature vector is similar to the feature vector of the model 3D point pair. Figure 1A and 1B The feature vectors may be considered similar if they have similar values within the user of the system 100. For example, if the magnitudes of their distance component, first angle component, second angle component, third angle component, and third angle component are within a threshold value (given by Figure 1A and 1BIf the first eigenvector is substantially the same as the second eigenvector (set by a user of the system 100), then the first eigenvector is similar to the second eigenvector.
[0091] In box 608, the reference target 3D point pair can be aligned with the model 3D point pair. In box 610, other model 3D point pairs (such as point pair 604A) can be aligned with the reference target 3D point pair. In particular, the coordinates of the other model 3D point pairs can be multiplied by the translation matrix and the rotation matrix to orient the other model 3D point pairs to the coordinate system of the reference target 3D point pair. The translation matrix and the rotation matrix can constitute a six-dimensional pose hypothesis. In this way, various model 3D point pairs and various reference target 3D point pairs can be identified and aligned. The individual translation and rotation matrices used for alignment points can constitute other six-dimensional pose hypotheses. Therefore, multiple six-dimensional pose hypotheses can be generated. Although Fig. 6A and 6B describes a specific six-dimensional pose assumption. Although Fig. 6A and 6B One particular method 600 for generating a six-dimensional pose hypothesis is described, but other known methods may be used.
[0092] Figure 7 An exemplary method 700 for improving candidate six-dimensional pose hypotheses according to a specific embodiment of the present invention is described. As described below, some transformation model three-dimensional points of the candidate six-dimensional pose hypothesis may be obscured by other transformation model three-dimensional points of the candidate six-dimensional pose hypothesis. When provided as input to an iterative closest point (ICP) algorithm for further refinement, these obscured transformation model three-dimensional points can be extracted from the model three-dimensional point cloud data of the candidate six-dimensional pose hypothesis. By removing the obscured three-dimensional point cloud data, the computing resources required to perform ICP can be reduced. In addition, the accuracy and precision of the improved six-dimensional pose hypothesis are also improved.
[0093] According to the translation and rotation matrices associated with the candidate 6D pose hypothesis, the model 3D points of the model 3D point cloud data are translated and rotated to generate transformed model 3D points and transformed model 3D point cloud data. However, some transformed model 3D points associated with the candidate 6D pose hypothesis may obscure other transformed model 3D points associated with the candidate 6D pose hypothesis, as depicted in the transformed model 3D point cloud data 702 associated with the candidate 6D pose hypothesis. For example, in Figure 7 The transformed model 3D point in the transformed model 3D point cloud data 702 in the positive z direction of the coordinate system shown in FIG. Figure 7 The transformed model 3D points in the transformed model 3D point cloud data 702 in the negative z direction of the coordinate system are shown.
[0094] Therefore, at block 704, the first transformed model 3D point in the transformed model 3D point cloud data occludes the second transformed model 3D point in the transformed model 3D point cloud data, and the transformed model 3D point can be determined based on their relative positions to the imaging device (eg, Figure 1A The first conversion model 3D point is determined by the coordinates of the known coordinate system of the imaging device 114). At box 706, the second conversion model 3D point (i.e., the obscured conversion model 3D point) of the conversion model 3D point cloud data can be extracted, leaving the first conversion model 3D point. At box 708, iterative closest point (ICP) improvement can be performed based on the remaining conversion model 3D point cloud data (i.e., the first conversion model 3D point). In this way, the candidate six-dimensional pose hypothesis can be improved without incurring additional computational costs to perform ICP improvement on the second conversion model 3D point obscured by the first conversion model 3D point. Since the obscured conversion model 3D points may be insignificant (i.e., their inclusion in the ICP improvement will not enhance the candidate six-dimensional pose hypothesis), the obscured model 3D points can be deleted to save computing resources. In addition, by applying the method 700 of the present invention, the accuracy and precision of the improved candidate six-dimensional pose hypothesis can be improved.
[0095] Figure 1A and 1B Any process shown or described in any system, Figure 2A-1 , 2A-2 , the methods of 2B-1, 2B-2, 3, 6A, 6B, 6B and 7, Figure 4A , 4B , 5A and 5B, or a combination thereof, may be controlled by a processing unit, such as a central processing unit (CPU), a controller, a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), other hardware devices, firmware devices, or any combination thereof. As an example, Figure 1A and 1B Any system in Figure 2A-1 , 2A-2 , the methods of 2B-1, 2B-2, 3, 6A, 6B, 6B and 7, Figure 4A , 4B , 5A and 5B, or a combination thereof, may be executed by one or more processors that perform object recognition. In addition, Figure 1A and 1B In a system, Figure 2A-1 , 2A-2 , 2B-1, 2B-2, 3, 6A, 6B, 6B and 7, or Figure 4A , 4B , 5A and 5B, the first part of the data structure can be Figure 1A and 1B In another system, Figure 2A-1 ,2A-2 , 2B-1, 2B-2, 3, 6A, 6B, 6B and 7, or Figure 4A , 4B , 5A and 5B. For example, Figure 3 The first part of the method 300 may be related to Figure 7 The second part of method 700 is combined.
[0096] Although one or more of Figures 1-7 may illustrate a system, device, data structure, method, or combination thereof according to the teachings of the present disclosure, the present disclosure is not limited to the systems, devices, methods, or combinations thereof described herein. One or more functions or components in Figures 1-7 may be combined with one or more other parts of another one or more functions or components in Figures 1-7, and therefore, any of the embodiments described herein should not be interpreted as limiting, and the embodiments of the present disclosure may be appropriately combined without departing from the teachings of the present disclosure.
[0097] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
[0098] Those skilled in the art will further appreciate that various illustrative logic blocks, modules, circuits and algorithm blocks related to the present disclosure can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of this hardware and software, the functions of various illustrative components, blocks, modules, circuits and steps have been generally described above. This function is implemented in the form of hardware or software, depending on the specific application and the design restrictions to the entire system. Those skilled in the art can implement the functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing deviations from the scope of the present disclosure. Those skilled in the art will also be easily aware that the order or combination of components, methods or interactions described here are merely exemplary, and components, methods or interactions of various aspects of the present disclosure can be combined or performed in a manner other than illustrated and described.
[0099] The functional blocks and modules in FIG. 1 may include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software codes, firmware codes, etc., or any combination thereof. Consistent with the foregoing, the various exemplary logic blocks, modules, and circuits disclosed herein may be implemented or executed by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in another case, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration.
[0100] The steps of the method or algorithm related to the present disclosure can be directly embodied in hardware, or in a software module executed by a processor, or a combination of the two. The software module can reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM or any other known form of storage medium. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. In another case, the storage medium can be integrated with the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal, a base station, a sensor or any other communication device. In an alternative, the processor and the storage medium can reside in a user terminal as discrete components.
[0101] In one or more exemplary designs, the functions may be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Computer-readable storage media may be any available medium that can be accessed by a general-purpose computer or a special-purpose computer. By way of example and not limitation, such a computer-readable storage medium may include RAM, ROM, EEPROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the required program code means in the form of an instruction or data structure, and can be accessed by a general-purpose computer or a special-purpose computer, or a general-purpose or special-purpose processor. In addition, the connection may be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber-optic cable, a twisted pair, or a digital subscriber line (DSL), the coaxial cable, the fiber-optic cable, the twisted pair, or the DSL are included in the definition of the medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically and discs reproduce data with lasers. Combinations of the above should also be included within the scope of computer readable media.
[0102] Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and modifications may be made herein without departing from the spirit and scope of the present invention as defined by the appended claims. In addition, the scope of the present application is not limited to the specific embodiments of the processes, machines, manufactures, material compositions, means, methods and steps described in the specification. It will be easily appreciated by those of ordinary skill in the art from the disclosure of the present invention that the processes, machines, manufactures, material compositions, means, methods or steps that currently exist or will be developed to perform substantially the same functions can achieve substantially the same results as the corresponding embodiments described herein according to the present invention, and accordingly, the appended claims are intended to include such processes, machines, manufactures, material compositions, means, methods or steps within their scope.
Claims
1. A method for selecting a six-dimensional pose hypothesis from a plurality of six-dimensional pose hypotheses, the method comprising: using the plurality of six-dimensional pose hypotheses to convert model three-dimensional point cloud data into a plurality of transformed model three-dimensional point cloud data instances, wherein the model three-dimensional point cloud data is derived from a three-dimensional representation of an object to be identified in a plurality of object images, wherein the plurality of transformed model three-dimensional point cloud data instances are converted in parallel from the model three-dimensional point cloud data; Based on determining that the model 3D point cloud data is not converted by one or more six-dimensional pose hypotheses among the multiple six-dimensional pose hypotheses, converting the model 3D point cloud data into the multiple converted model 3D point cloud data instances by using the multiple six-dimensional pose hypotheses for one or more iterations; and Selecting a candidate six-dimensional pose hypothesis from the multiple six-dimensional pose hypotheses according to an overlap degree, wherein the overlap degree is an overlap degree between a conversion model three-dimensional point cloud data instance generated by the candidate six-dimensional pose hypothesis and a target three-dimensional point of the target three-dimensional point cloud data obtained from the multiple object images; in, The method of converting model three-dimensional point cloud data into multiple transformed model three-dimensional point cloud data instances using multiple six-dimensional pose hypotheses includes converting each model three-dimensional point of each copy of the model three-dimensional point cloud data in parallel into a transformed model three-dimensional point of a transformed model three-dimensional point cloud data instance in the multiple transformed model three-dimensional point cloud data instances, and wherein each transformed model three-dimensional point cloud instance includes: a model three-dimensional point of the model three-dimensional point cloud data that has been translated and rotated by a translation and rotation matrix corresponding to each six-dimensional pose hypothesis in the multiple six-dimensional pose hypotheses.
2. The method according to claim 1, wherein the converting model three-dimensional point cloud comprises: Parallel mathematical operations are performed on the model three-dimensional point cloud data, wherein each six-dimensional pose hypothesis of the multiple six-dimensional pose hypotheses corresponds to a set of mathematical operations; wherein the mathematical operations include: multiplying each coordinate of each point constituting a first copy of the model three-dimensional point cloud data by a translation matrix corresponding to the first six-dimensional pose hypothesis and a rotation matrix corresponding to the first six-dimensional pose hypothesis.
3. The method according to claim 1, further comprising: Determining a plurality of overlap measures between a transformed model 3D point of each transformed model 3D point cloud instance and a target 3D point of the target 3D point cloud data; determining an overlap metric having a maximum magnitude among the plurality of overlap metrics; as well as A six-dimensional pose hypothesis corresponding to an overlap metric having a maximum magnitude is selected from the plurality of six-dimensional pose hypotheses, wherein the selected six-dimensional pose hypothesis comprises the candidate six-dimensional pose hypothesis.
4. The method of claim 3, wherein determining a plurality of overlap metrics comprises: The distance between the converted model 3D point of the converted model 3D point cloud data instance and the target 3D point of the target 3D point cloud data is determined.
5. The method according to claim 4, further comprising: Partitioning the target three-dimensional point cloud data; Assigning a unique partition number to each partition of the partitioned target three-dimensional point cloud data; Based at least in part on the position of the transformed model 3D point within the partition, a distance between the transformed model 3D point of the transformed model 3D point cloud instance and a target 3D point corresponding to the unique partition number is determined.
6. The method according to claim 1, further comprising: After selecting the candidate six-dimensional pose hypothesis, improving the candidate six-dimensional pose hypothesis by extracting the obscured three-dimensional points of the transformation model; The masked transformation model 3D point corresponds to the transformation model 3D point cloud instance associated with the candidate 6D pose hypothesis; as well as An iterative closest point (ICP) algorithm is performed on the conversion model 3D points of the conversion model 3D point cloud instance corresponding to the candidate 6D pose hypothesis and the target 3D points of the target 3D point data.
7. The method according to claim 1, further comprising: Sending candidate six-dimensional pose hypothesis data corresponding to the candidate six-dimensional pose hypothesis to a robot, wherein the robot is configured to recognize the object based on the candidate six-dimensional pose hypothesis data.
8. The method according to claim 1, further comprising: One or more copies of the model three-dimensional point cloud data are stored in a one-dimensional array to facilitate conversion of the model three-dimensional point cloud data into the plurality of converted model three-dimensional point cloud data instances.
9. An apparatus for selecting a six-dimensional pose hypothesis from a plurality of six-dimensional pose hypotheses, comprising: A memory configured to store: Model three-dimensional point cloud data corresponding to a three-dimensional representation of an object to be identified in the image of the plurality of objects; Six-dimensional pose hypothesis data, which corresponds to a plurality of six-dimensional pose hypotheses; as well as target three-dimensional point cloud data corresponding to the image; and a processor configured to: Generating a plurality of transformed model three-dimensional point cloud data instances by performing parallel mathematical operations on the model three-dimensional point cloud data, wherein the mathematical operations correspond to the six-dimensional pose hypothesis data; Determine a plurality of overlap metrics, each overlap metric of the plurality of overlap metrics corresponding to each transformed model 3D point cloud data instance, the overlap metric representing a degree of overlap between a transformed model 3D point of the transformed model 3D point cloud data instance and a target model 3D point of the target model 3D point cloud data; as well as Selecting a six-dimensional pose hypothesis corresponding to a maximum overlap measure among the multiple overlap measures as a candidate six-dimensional pose hypothesis; The mathematical operation includes: multiplying each coordinate of each point constituting the first copy of the three-dimensional point cloud data of the model by a translation matrix corresponding to the first six-dimensional pose hypothesis and a rotation matrix corresponding to the first six-dimensional pose hypothesis; Each transformed model three-dimensional point cloud instance includes: model three-dimensional points of the model three-dimensional point cloud data that have been translated and rotated by the translation and rotation matrices corresponding to each six-dimensional pose hypothesis in the multiple six-dimensional pose hypotheses.
10. The apparatus according to claim 9, wherein each of the plurality of six-dimensional pose hypotheses corresponds to a translation matrix and a rotation matrix, wherein the six-dimensional pose hypothesis data corresponds to a plurality of translation matrices and rotation matrices.
11. The apparatus of claim 10, wherein the processor being configured to determine the plurality of overlap metrics further comprises the processor being configured to: Determine a distance between a target 3D point of the target 3D point cloud data and a transformed model 3D point of each transformed model 3D point cloud data instance; comparing the distance to a threshold value stored in the memory; In response to the distance being less than the threshold, a variable stored in the memory is incremented, the variable corresponding to an overlap metric.
12. The apparatus of claim 11, wherein the processor is further configured to: comparing the magnitude of each overlap metric in the plurality of overlap metrics; The overlap metric having the largest magnitude is determined as the largest overlap metric.
13. The apparatus according to claim 10, further comprising: an imaging device configured to generate a three-dimensional image of the plurality of objects; and A network interface is configured to send candidate six-dimensional pose hypothesis data corresponding to the candidate six-dimensional pose hypothesis to a second device, wherein the candidate six-dimensional pose hypothesis data is configured to enable the second device to recognize the object from the images of the plurality of objects.
14. The device according to claim 13, wherein the second device is a robot, wherein the six-dimensional pose hypothesis data is combined with control instructions and is further configured to enable the robot to grasp the object.
15. The apparatus of claim 9, further comprising a robotic arm configured to grasp the object based at least in part on candidate six-dimensional pose hypothesis data corresponding to the candidate six-dimensional pose hypothesis.
16. A non-transitory computer readable medium comprising instructions which, when executed by a processor, cause the processor to: A plurality of transformed model three-dimensional point cloud data instances are generated by performing parallel mathematical operations on the model three-dimensional point cloud data stored in a one-dimensional array, each mathematical operation of the mathematical operations corresponding to a six-dimensional pose hypothesis among a plurality of six-dimensional pose hypotheses, wherein: The model three-dimensional point cloud data is extracted from the three-dimensional representation of the object to be identified; as well as Selecting a candidate six-dimensional pose hypothesis from the plurality of six-dimensional pose hypotheses, wherein the candidate six-dimensional pose hypothesis is selected based on an overlap metric; wherein the instructions further cause the processor to determine a plurality of overlap metrics, each overlap metric of the plurality of overlap metrics indicating a degree of overlap between a transformed model 3D point of each transformed model 3D point cloud data instance of the plurality of transformed model 3D point cloud data instances and a target 3D point of the target 3D point cloud data extracted from the plurality of object images; The mathematical operation includes: multiplying each coordinate of each point constituting the first copy of the three-dimensional point cloud data of the model by a translation matrix corresponding to the first six-dimensional pose hypothesis and a rotation matrix corresponding to the first six-dimensional pose hypothesis; Each transformed model 3D point cloud instance includes: model 3D points of the model 3D point cloud data that have been translated and rotated by the translation and rotation matrices corresponding to each 6-D pose hypothesis of the plurality of 6-D pose hypotheses.
17. The non-transitory computer-readable medium of claim 16, wherein the target three-dimensional point corresponds to a partition number applied to the target three-dimensional point cloud data, and wherein the instructions further cause the processor to partition the target three-dimensional point cloud data.
18. The non-transitory computer readable medium of claim 16, wherein the instructions further cause the processor to position and orient a robotic arm of a device to grasp the object.
Citation Information
Patent Citations
Device and method for recognizing three-dimensional position and orientation of article
CN103085076A
Measuring method for foot three-dimensional foot-type information and three-dimensional reconstruction model by means of RGB-D camera
CN103971409A
Single image scene six-degree-of-freedom positioning method of adjacent pose fusion guidance
CN105956074A
Method and system for identifying and positioning targets on basis of three-dimensional point cloud
CN106778790A
Camera pose estimation method oriented to RGBD (Red, Green and Blue-Depth) data stream
CN106780576A
