Machine vision image data processing method and device
By converting 3D data into a densely packed lattice with representative vectors, the method addresses inefficiencies in existing machine vision techniques, enhancing processing speed and accuracy in pattern recognition.
Patent Information
- Application Number
- CN202410570546.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-09-12
- Filing Date
- 2019-09-12
- Publication Date
- 2024-12-03
AI Technical Summary
Existing machine vision techniques face inefficiencies in processing three-dimensional data, particularly in searching for neighboring points in point clouds, which significantly reduces processing time and parallelization efficiency.
The method involves converting three-dimensional data into a densely packed lattice or field, where each cell contains representative data such as vectors, allowing for serial processing and avoiding the need to search for neighboring points, thereby improving processing time and efficiency.
This approach enhances the speed and accuracy of pattern recognition in three-dimensional data by reducing the time-consuming branches that interrupt parallelization, thus improving overall performance.
Smart Images

Figure CN119068167A_ABST
Abstract
Description
[0001] Divisional application
[0002] This application is a divisional application of application number 201910865951.8, application date September 12, 2019, and titled “Machine Vision Image Data Processing Method and Device”. Technical Field
[0003] The technology described herein generally relates to methods and apparatus for machine vision, including techniques for processing image data and for searching for patterns in images. Background Art
[0004] One task that machine vision systems commonly perform is to attempt to search for and identify the location and orientation of a pattern of interest in an image. Some techniques use a model to represent the pattern of interest, which can include multiple probes. Each probe is a point of interest and associated data (e.g., a location and a vector). For example, each probe can be used to determine a measure of similarity between a runtime image feature or region and a pattern feature or region at a specific location. Multiple probes can be applied to a runtime image in multiple poses, and information from the probes in each pose can be used to determine the most likely pose of the pattern in the runtime image.
[0005] In order to speed up the pattern recognition process, some techniques use a multi-step approach to the pattern search process. For example, the first step may include a rough search, trying to find one or more approximate areas in the image that may contain the pattern. The second step (and / or multiple other steps) can be used to optimize the search by searching each of the one or more approximate areas. For example, the algorithm can use multiple different models, where the system uses each model to obtain a different associated resolution of the image. Therefore, in the pattern recognition process, a rough resolution and associated model can be used to initially identify a rough approximation of the pose of the pattern instance in the image. Thereafter, a relatively finer resolution and associated model can be used to more accurately identify the pose of the pattern instance in the image. This iterative process continues until the finest resolution model is used and the exact pose of the pattern instance is identified. Summary of the invention
[0006] According to the disclosed subject matter, apparatus, systems, and methods are provided for improving machine vision techniques, particularly for improving the speed and accuracy of searching for patterns in images.
[0007] Some aspects relate to a computerized method for testing a pose of a three-dimensional data model. The method includes: receiving three-dimensional data of an object, the three-dimensional data including a set of data input items; converting the three-dimensional data into a field including a set of cells (each having an associated value), the conversion process including determining representative data for each cell value based on one or more data input items in the set of data input items of the three-dimensional data; and testing the pose of the model using the field to determine a score for the pose.
[0008] In some examples, converting the three-dimensional data into a field includes generating a three-dimensional array of a set of values.
[0009] In some examples, converting the three-dimensional data into a field includes generating a densely packed lattice, wherein the densely packed lattice includes data for each cell of the lattice.
[0010] In some examples, the set of data inputs for the three-dimensional data includes a list of points; determining representative data for each cell value based on the one or more data inputs includes determining a vector based on one or more associated points in the list of points. Determining the vector of the one or more associated points includes determining an internal association of the cell with the object based on the list of points; determining the vector includes setting the vector to zero. Testing the pose of the model using a field to determine the score includes testing a set of probes of the model against the field to determine the score, including summing the dot products of each probe and the associated vector in the field.
[0011] In some examples, the three-dimensional data is converted into a field comprising a set of cells (each having an associated value), the conversion process including determining a representative vector for each cell value, which includes generating an accumulation matrix, the matrix including calculating the outer product of each vector of a set of vectors with itself, wherein the set of vectors is data determined based on one or more data input items from a set of data input items of the three-dimensional data; and extracting eigenvectors, eigenvalues, or both from the accumulation matrix to determine the representative vector.
[0012] Some aspects relate to a system for testing the pose of a three-dimensional data model. The system includes one or more processors for: receiving three-dimensional data of an object, the three-dimensional data including a set of data input items; converting the three-dimensional data into a field including a set of cells (each having an associated value), the conversion process including determining representative data for each cell value based on one or more data input items in the set of data input items of the three-dimensional data; and testing the pose of the model using the field to determine a score for the pose.
[0013] In some examples, converting the three-dimensional data into a field includes generating a three-dimensional array of a set of values.
[0014] In some examples, converting the three-dimensional data into a field includes generating a densely packed lattice, wherein the densely packed lattice includes data for each cell of the lattice.
[0015] In some examples, the set of data inputs for the three-dimensional data includes a list of points; determining representative data for each cell value based on the one or more data inputs includes determining a vector based on one or more associated points in the list of points. Determining the vector of the one or more associated points includes determining an internal association of the cell with the object based on the list of points; determining the vector includes setting the vector to zero. Testing the pose of the model using a field to determine a score includes testing a set of probes of the model against the field to determine the score, including summing a dot product of each probe and an associated vector in the field.
[0016] In some examples, the three-dimensional data is converted into a field comprising a set of cells (each having an associated value), the conversion process including determining a representative vector for each cell value, which includes generating an accumulation matrix, the matrix including calculating the outer product of each vector of a set of vectors with itself, wherein the set of vectors is data determined based on one or more data input items from a set of data input items of the three-dimensional data; and extracting eigenvectors, eigenvalues, or both from the accumulation matrix to determine the representative vector.
[0017] Some aspects relate to at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the actions of receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; converting the three-dimensional data into a field comprising a set of cells (each having an associated value), the conversion process comprising determining representative data for each cell value based on one or more data input items in the set of data input items of the three-dimensional data; and testing the pose of the model using the field to determine a score for the pose.
[0018] In some examples, converting the three-dimensional data into a field includes generating a three-dimensional array of a set of values.
[0019] In some examples, converting the three-dimensional data into a field includes generating a densely packed lattice, wherein the densely packed lattice includes data for each cell of the lattice.
[0020] In some examples, the set of data input items of the three-dimensional data includes a list of points; and determining representative data for each cell value includes determining a vector based on one or more associated points in the list of points.
[0021] In some examples, determining the vector of the one or more associated points includes determining an internal association of the cell with the object based on the point list; and determining the vector includes setting the vector to zero.
[0022] In some examples, the three-dimensional data is converted into a field comprising a set of cells (each having an associated value), the conversion process including determining a representative vector for each cell value, which includes generating an accumulation matrix, the matrix including calculating the outer product of each vector of a set of vectors with itself, wherein the set of vectors is data determined based on one or more data input items from a set of data input items of the three-dimensional data; and extracting eigenvectors, eigenvalues, or both from the accumulation matrix to determine the representative vector.
[0023] Some aspects relate to a computerized method for testing the pose of a model of image data. The method includes receiving image data of an object, the image data including a set of data entries. The method includes determining a set of regions of the image data, wherein each region of the set of regions includes an associated set of adjacent data entries in the set of data entries. The method includes generating processed image data, wherein the processed image data includes a set of cells, each cell having an associated value; generating the processed image data includes determining, for each region of the set of regions, a maximum possible score for each data entry in an associated set of adjacent data entries in the image data. The method includes setting one or more values of the set of values based on the determined maximum possible scores. The method includes testing the pose of the model using the processed image data.
[0024] In some examples, receiving image data includes receiving 2D image data, wherein each data input item includes a 2D vector; determining the maximum possible score for each processed image data value in the set of values includes determining a scalar value based on the 2D vector in the value associated region.
[0025] In some examples, testing the model pose using the processed data includes: determining that the pose score does not exceed a predetermined threshold, the process comprising testing multiple probes of the model on an associated scalar value of the processed data; and eliminating a set of poses associated with each region for determining an associated scalar value from further testing.
[0026] In some examples, receiving image data includes receiving 3D image data, wherein each data entry includes a 3D vector; determining a maximum possible score for each processed image data value of the set of values includes determining a scalar value based on the 3D vectors in the region associated with the value. Testing the pose of the model using the processed data may include: determining that the pose score does not exceed a predetermined threshold, the process including testing a plurality of probes of the model against the scalar value associated with the processed data; and eliminating a set of poses associated with each region for determining the associated scalar value from further testing.
[0027] In some examples, the method includes: converting the three-dimensional data into a second field including a second set of cells (each associated with a second value), the conversion process including determining representative data for each second cell value based on one or more data input items in a set of data input items for the three-dimensional data; and based on testing the model posture using the field, using the second field to test the posture of the model.
[0028] Some aspects relate to a system for testing a pose of a model against image data, the system comprising one or more processors for receiving image data of an object, the image data comprising a set of data entries. The one or more processors are used to determine a set of regions of the image data, wherein each region of the set of regions comprises an associated set of adjacent data entries in the set of data entries. The one or more processors are used to generate processed image data, wherein the processed image data comprises a set of cells, each cell having an associated value; generating the processed image data comprises determining, for each region of the set of regions, a maximum possible score for each data entry in a set of adjacent data entries associated in the image data. The one or more processors are used to set one or more values of the set of values based on the determined maximum possible scores. The one or more processors are used to test the pose of the model using the processed image data.
[0029] In some examples, receiving image data includes receiving 2D image data, wherein each data entry includes a 2D vector; determining a maximum possible score for each processed image data value in the set of values includes determining a scalar value based on the 2D vector in the region associated with the value. Testing the pose of the model using the processed data may include: determining that the pose score does not exceed a predetermined threshold, the process including testing a plurality of probes of the model against the scalar value associated with the processed data; and eliminating a set of poses associated with each region for determining the associated scalar value from further testing.
[0030] In some examples, receiving image data includes receiving 3D image data, wherein each data entry includes a 3D vector; determining a maximum possible score for each processed image data value in the set of values includes determining a scalar value based on the 3D vectors in the region associated with the value. Testing the pose of the model using the processed data may include: determining that the pose score does not exceed a predetermined threshold, the process including testing a plurality of probes of the model against the scalar value associated with the processed data; and eliminating a set of poses associated with each region for determining the associated scalar value from further testing.
[0031] In some examples, the one or more processors are also used to convert the three-dimensional data into a second field comprising a second group of cells (each associated with a second value), the conversion process comprising determining representative data for each second cell value based on one or more data input items in a group of data input items for the three-dimensional data; and based on testing the model posture using the field, using the second field to test the posture of the model.
[0032] Some aspects relate to at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the following actions: receive three-dimensional data of an object, the three-dimensional data comprising a set of data entries. The instructions cause the at least one computer hardware processor to determine a set of regions of the image data, wherein each region of the set of regions comprises an associated set of adjacent data entries in the set of data entries. The instructions cause the at least one computer hardware processor to generate processed image data, wherein the processed image data comprises a set of cells, each cell having an associated value; generating the processed image data comprises determining, for each region of the set of regions, a maximum possible score for each data entry in an associated set of adjacent data entries in the image data. The instructions cause the at least one computer hardware processor to set one or more values of the set of values based on the determined maximum possible scores. The instructions cause the at least one computer hardware processor to test a pose of the model using the processed image data.
[0033] In some examples, the instructions cause the at least one computer hardware processor to perform the following actions: receiving image data, including 2D image data, wherein each data entry comprises a 2D vector; determining a maximum possible score for each processed image data value in the set of values comprises determining a scalar value based on the 2D vectors in the region associated with the value. Testing the pose of the model using the processed data may include: determining that the pose score does not exceed a predetermined threshold, the process comprising testing a plurality of probes of the model against the scalar value associated with the processed data; and eliminating a set of poses associated with each region for determining the associated scalar value from further testing.
[0034] In some examples, receiving image data includes receiving 3D image data, wherein each data entry includes a 3D vector; determining a maximum possible score for each processed image data value of the set of values includes determining a scalar value based on the 3D vectors in a region associated with the values.
[0035] In some examples, testing the model pose using the processed data includes: determining that the pose score does not exceed a predetermined threshold, the process comprising testing multiple probes of the model on an associated scalar value of the processed data; and eliminating a set of poses associated with each region for determining an associated scalar value from further testing.
[0036] In some examples, the instructions cause the at least one computer hardware processor to perform the following actions: converting the three-dimensional data into a second field including a second set of cells (each associated with a second value), the conversion process including determining representative data for each second cell value based on one or more data input items in a set of data input items for the three-dimensional data; and using the second field to test the pose of the model based on testing the pose of the model using the field.
[0037] Some aspects relate to a computerized method for testing the pose of a three-dimensional model, the method comprising: storing a three-dimensional model, the three-dimensional model comprising a set of probes; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; converting the three-dimensional data into a set of fields, comprising: generating a first field comprising a first set of values, wherein each value of the first set of values indicates a first feature of one or more data input items associated with the set of data input items; generating a second field comprising a second set of values, wherein each second value in the second set of values indicates a second feature of one or more data input items associated with the data input items, wherein the second feature is different from the first feature; and testing the pose of the three-dimensional model using the set of fields, comprising testing the set of probes against the set of fields to determine a score for the pose.
[0038] In some examples, generating the first field and the second field includes generating a three-dimensional array for each field, wherein each three-dimensional array includes a set of three indexes, which include an index for each dimension; each three-dimensional array uses the set of three indexes to represent the x, y and z positions of the associated first value and second value.
[0039] In some examples, the probe, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, intensity data, or some combination thereof.
[0040] In some examples, testing the gesture to determine a score for the gesture includes summing a dot product of each probe and the associated value.
[0041] In some examples, the method includes testing a plurality of poses to determine a plurality of association scores; determining which of the plurality of poses includes generating a set of poses with scores above a predetermined threshold; storing the set of poses for subsequent processing. Each pose in the set of poses represents a local peak of the association score, and the method further includes refining the set of poses to determine a top pose of the model.
[0042] Some aspects relate to a system for determining image acquisition parameters, the system comprising one or more processors for: storing a three-dimensional model, the three-dimensional model comprising a set of probes; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; converting the three-dimensional data into a set of fields, the process comprising: generating a first field comprising a first set of values, wherein each value of the first set of values indicates a first feature of one or more data input items associated with the set of data input items; generating a second field comprising a second set of values, wherein each second value in the second set of values indicates a second feature of one or more data input items associated with the data input items, wherein the second feature is different from the first feature; using the set of fields to test the posture of the three-dimensional model, including testing the set of probes on the set of fields to determine a score for the posture.
[0043] In some examples, generating the first field and the second field includes generating a three-dimensional array for each field, wherein each three-dimensional array includes a set of three indexes, which include an index for each dimension; each three-dimensional array uses the set of three indexes to represent the x, y and z positions of the associated first value and second value.
[0044] In some examples, the probe, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, intensity data, or some combination thereof.
[0045] In some examples, testing the gesture to determine a score for the gesture includes summing a dot product of each probe and the associated value.
[0046] In some examples, the one or more processors are further configured to test multiple gestures to determine multiple association scores; determine which gestures of the multiple gestures include generating a set of gestures using scores above a predetermined threshold; and store the set of gestures for subsequent processing.
[0047] In some examples, each pose in the set of poses represents a local peak of the association score, and the method further includes refining the set of poses to determine a top pose of the model.
[0048] Some embodiments relate to at least one non-transitory computer-readable storage medium storing processor-executable instructions, which, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the following actions: storing a three-dimensional model, the three-dimensional model comprising a set of probes; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; converting the three-dimensional data into a set of fields, including generating a first field comprising a first set of values, wherein each value of the first set of values indicates a first feature of one or more data input items associated with the set of data input items; generating a second field comprising a second set of values, wherein each second value in the second set of values indicates a second feature of one or more data input items associated with the data input items, wherein the second feature is different from the first feature; and testing the posture of the three-dimensional model using the set of fields, including testing the set of probes on the set of fields to determine a score for the posture.
[0049] In some examples, generating the first field and the second field includes generating a three-dimensional array for each field, wherein each three-dimensional array includes a set of three indexes, which include an index for each dimension; each three-dimensional array uses the set of three indexes to represent the x, y and z positions of the associated first value and second value.
[0050] In some examples, the probe, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, intensity data, or some combination thereof.
[0051] In some examples, testing the gesture to determine a score for the gesture includes summing a dot product of each probe and the associated value.
[0052] In some examples, the instructions further cause the one or more processors to test a plurality of gestures to determine a plurality of association scores; determine which of the plurality of gestures include generating a set of gestures using scores above a predetermined threshold; and store the set of gestures for subsequent processing.
[0053] In some examples, each pose in the set of poses represents a local peak of the association score, and the instructions further include refining the set of poses to determine a top pose of the model.
[0054] Therefore, the disclosed subject matter has been discussed quite extensively in order to better understand the following detailed description and to better appreciate the contribution that the present invention makes to the art. Of course, other features of the disclosed subject matter will also be described below, which will form the subject matter of the appended claims. It should be understood that the words and terms used herein are for descriptive purposes and should not be regarded as limiting. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In the drawings, each identical or nearly identical component shown in each figure is represented by the same reference character. For clarity, not every component may be labeled in each figure. The drawings are not necessarily drawn to scale, but rather focus is placed on various aspects of the techniques and devices described herein.
[0056] Figure 1 An exemplary machine vision system is shown in accordance with some embodiments.
[0057] Figure 2A An exemplary method for generating fields and testing a model using the fields is shown in accordance with some embodiments.
[0058] Figure 2B An exemplary method for generating fields according to some embodiments is shown.
[0059] Figure 3 Exemplary three-dimensional image data and associated fields are shown in accordance with some embodiments.
[0060] Figure 4 An exemplary method for generating processed image data for a coarse search is shown in accordance with some embodiments.
[0061] Figure 5 Exemplary two-dimensional image data and associated processed image data are shown in accordance with some embodiments.
[0062] Figure 6 A set of poses and associated data entries for exemplary elimination of the two-dimensional image data during a coarse search is shown in accordance with some embodiments.
[0063] Figure 7 Exemplary three-dimensional image data and associated processed image data are shown in accordance with some embodiments.
[0064] Figure 8A set of poses and associated data entries for exemplary elimination of the three-dimensional image data during a coarse search is shown in accordance with some embodiments.
[0065] Fig. 9 An example method of initial pose searching in accordance with some embodiments is shown. DETAILED DESCRIPTION
[0066] In the following description, the system and method of the disclosed subject matter and the environment in which such system and method can operate are described in detail to provide a thorough understanding of the disclosed subject matter. In addition, it should be understood that the examples provided below are exemplary, and it is expected that the disclosed subject matter also includes other systems and methods.
[0067] Figure 1 An exemplary machine vision system 100 is shown in accordance with some embodiments. The exemplary machine vision system 100 includes a camera 102 (or other imaging acquisition device) and a computer 104. The computer 104 includes one or more processors and a human-machine interface in the form of a computer display and optionally one or more input devices (e.g., a keyboard, a mouse, a trackball, etc.). The camera 102 also includes a lens 106 and a camera sensor element (not shown). The lens 106 includes a field of view 108, which focuses light from the field of view 108 onto the sensor element. The sensor element generates a digital image of the camera field of view 108 and provides the image to a processor that forms part of the computer 104. As shown in FIG. Figure 1 As shown in the example of FIG. 1 , an object 112 enters the field of view 108 of the camera 102 along the conveyor 110. As further discussed herein, the camera 102 can generate one or more digital images while the object 112 is being processed in the field of view 108. In operation, the conveyor can contain multiple objects. These objects can, for example, pass through the field of view 108 of the camera 102 in sequence during the inspection process. Therefore, the camera 102 can obtain at least one image of each observed object 112.
[0068] In some embodiments, the camera 102 is a two-dimensional imaging device, such as a two-dimensional (2D) CCD or CMOS imaging array. In some embodiments, the two-dimensional imaging device generates a two-dimensional array of brightness values. In some embodiments, the machine vision system can process 2D data by generating a two-dimensional gradient field image, etc. For example, a gradient field image may include a set of cells with associated lengths and directions. For example, a gradient field may contain Cartesian components of a vector (x, y), which may represent length and direction, and a gradient field may store actual (r, theta) values, etc. In some embodiments, the camera 103 is a three-dimensional (3D) imaging device. A 3D imaging device can generate a set of (x, y, z) points (e.g., the z axis adds a third dimension, such as the distance from the 3D imaging device). A 3D imaging device can use various 3D image generation techniques, such as shading recovery shape, stereo imaging, time of flight technology, projector-based technology, and / or other 3D generation technologies.
[0069] In some embodiments, the machine vision system processes 3D data from the camera 103. The 3D data received from the camera 103 may include point clouds and / or range images, etc. The point cloud may include a set of 3D points located at or near the surface of a solid object. For example, these points may be represented by their coordinates in a rectilinear coordinate system or other coordinate systems. In some embodiments, other information (e.g., a mesh or grid structure indicating adjacent points on the surface of an object) may also be present selectively. In some embodiments, information about surface features may be included in the input point cloud, including curvature, surface normals, edges, and / or color and albedo information (derived from sensor measurements or previously calculated information). In some embodiments, 2D and / or 3D data may be obtained from 2D and / or 3D sensors, CAD or other solid models, and / or by preprocessing range images, 2D images, and / or other images.
[0070] Examples of computer 104 may include, but are not limited to, a single server computer, a series of server computers, a single personal computer, a series of personal computers, a minicomputer, a mainframe computer, and / or a computing cloud. Various components of computer 104 may execute one or more operating systems, including, but not limited to: Microsoft Windows Server 2003, ... and Microsoft Windows Server 2003. TM 、NovellNetware TM 、Redhat Linux TM, Unix, and / or custom operating systems. One or more processors of computer 104 may be used to process operations stored in memory connected to the one or more processors. The memory may include, but is not limited to, hard drives, flash drives, tape drives, optical drives, RAID arrays, random access memory (RAM), and read-only memory (ROM).
[0071] As described herein, to search for a model in an image of an object, these techniques can be used to perform two (or more) stages, including a first stage of determining an approximate or rough location of the model in the image, and a second stage of refining the rough location to determine the specific location of the model. Using multiple stages is beneficial, for example, because different stages can use different technical approaches to perform the search to improve search speed, efficiency, etc. For example, for a 2D pattern search method, these techniques can include training a model of an object comprising a set of probes. Each probe in the 2D probes can include an (x, y) position and orientation. The machine vision system stores the trained model for use in searching subsequently captured runtime images of the field scene (e.g., Figure 1110 along the conveyor belt 110). The machine vision system can capture an image of the field scene, such as a 2D image with pixel values that represent the light accumulated in each cell. The machine vision system can convert the 2D image into a dense gradient field, where each cell includes a single gradient value that reflects the change in specific image data (for example, for constant areas, the gradient may be close to zero, and the edges are larger). The gradient value can be a vector, a scalar value (such as a length or an angle of orientation), etc. For example, if each cell contains a vector, the length of the vector can be zero. The machine vision system can test different transformations of the 2D model (for example, translation, rotation, scaling, skew, orientation, perspective, etc.) to calculate a score for a specific position. The score can reflect, among other things, the degree to which the probe matches the vector of the 2D image (for example, the vector is the gradient of the input 2D information, as further discussed herein). For example, some techniques can test the model one by one and obtain the highest scoring position for subsequent improvement. Some techniques can be used to focus on local peaks, because nearby searches will be similar, and the system can try to avoid marking all nearby matches as candidate states. For example, in a 3D environment, the image can be a range image, a point cloud, etc. As a general matter, as further discussed herein, a machine vision system can train a 3D model of an object comprising a set of probes. The machine vision system can use the 3D model to search for the pose of the 3D model in a running 3D image. As described herein, the machine vision system can process the 3D image to generate one or more 3D fields to facilitate searching for the pose of the 3D model. The fields can include a set of vectors. The Cartesian components of the vector (x, y, z) can be stored, which can represent length and direction, the fields can store actual (r, lat, long) values, etc.
[0072] The embodiments discussed herein can be used in a variety of different applications, some of which may include, but are not limited to, part picking in vision-guided robotics, three-dimensional inspection, automotive accessories, molded plastics and casting volume inspection, and assembly inspection. Such applications may include searching and identifying the position and orientation of a pattern of interest in an image (e.g., guiding a robotic gripper or inspecting an object). In some embodiments, a training step is used to develop a model to represent a pattern of interest, which may include multiple probes. Each probe is a point of interest and associated data (e.g., position and vector), and can be used to determine, among other things, the similarity of a runtime image feature or region to a pattern feature or region at a specific location. Multiple probes can be applied to a runtime image in multiple poses, and information from the probes in each pose can be used to determine the most likely pose of a pattern in the runtime image.
[0073] The inventors have found that existing machine vision techniques may have significant inefficiencies when using traditional 3D data. In particular, the inventors have found that a large amount of processing time is often consumed by searching for neighbors of a point in the 3D data (e.g., by searching for nearby points in a point cloud). For example, although machine vision systems can efficiently process continuous data, the efficiency of machine vision systems in searching and randomly accessing data is greatly reduced. In particular, computing devices typically include optimized hardware for massively parallelizing tasks repeated on continuous memory locations. Interrupting such parallelism using conditional branches may significantly reduce performance (e.g., because branches typically require stopping parallel activities, spending time executing branches / jumps, and then restarting parallel activities). The inventors have developed technical improvements to machine vision techniques to address these and other inefficiencies. As further discussed herein, these techniques include developing dense fields from 3D data, wherein the dense fields include data for each field value determined based on the 3D data. The inventors have found that because each value includes data, machine vision techniques can use field values as part of the process to avoid searching for neighboring points as described above, which can significantly reduce the processing time of existing machine vision techniques. The techniques disclosed herein can significantly improve performance by creating data at each input item of a field or grid and processing the data serially, thereby avoiding time-consuming branches that interrupt parallelization.
[0074] Figure 2A An exemplary method 200 for generating fields from 3D images to test 3D models is shown in accordance with some embodiments. In step 202, a machine vision system receives three-dimensional data (e.g., a point cloud, a dense 3D image, etc.) of an object. The three-dimensional data includes a set of data input items, such as a range image (e.g., each cell of which has a height), a point cloud, and / or other information about the 3D data. In step 204, the machine vision system converts the three-dimensional data into a field. The field contains a set of values. The method includes determining representative data derived from the 3D data for each value of the field. The representative data may include vectors and / or other representative information of the data input items, such as only vector lengths and / or directions, rather than complete vectors, matrices (e.g., vv discussed further herein), and the like. T) and the like. The method may include determining representative data from a set of data input items of three-dimensional data based on one or more data input items. In step 206, the machine vision system uses the field to test the pose of the model to determine the score of the pose. In step 208, the machine vision system determines whether the pose is acceptable or should be skipped (e.g., ignored, discarded, etc.). For example, the machine vision system can score the pose of the 3D model to the field and determine whether the score is above a threshold. As another example, as further discussed herein, the machine vision system can score the pose of the 3D model to multiple fields. If the score meets the threshold (e.g., above the threshold), the method continues to step 210, and the machine vision system stores the pose (e.g., for subsequent refinement, as discussed herein). If the score does not meet the threshold, the method continues to step 212 and determines whether there are other 3D model poses to be tested. If there are more poses, the method continues to return to step 206 to test another pose of the 3D model. If there are no more poses, the method goes to step 214 and ends.
[0075] Referring to step 204, converting the three-dimensional data to a field may include generating a three-dimensional array of a set of values. For example, the three dimensions may represent the x, y, and z axes of the 3D data. Each value in the value set may be a vector. A vector may be represented in various ways. For example, in some embodiments, each vector may be stored as an x, y, and z component, wherein each component may be represented using a certain number of bits (e.g., a signed 8-bit integer), wherein the value may be between -127 and 127. As another example, in some embodiments, each vector may be represented using a length and two angles. In some embodiments, converting the three-dimensional data to a field may include generating a densely filled grid. The densely filled grid may include, for example, the value of each possible point in the grid. The points in the grid may or may not be connected to the original position of the data entry in the 3D data. For example, a kernel or filter may be used so that the grid points have a grid different from that of the 3D data. This process may include converting a point cloud into a dense field, wherein the dense field includes a vector at each possible position of the dense field.
[0076] In some embodiments, these techniques may include applying a transformation to the 3D data before generating the field. For example, these techniques may include applying a rigid transformation, a linear transformation, and / or a nonlinear transformation before generating the field. In some embodiments, these techniques may include applying one or more transformations to account for distortion. For example, a point cloud may be acquired with a delay so that the values are skewed on one or more axes, and the point cloud may be transformed to account for the skew. In some embodiments, these techniques may include searching for distortion of the model when searching for the pose of the model. For example, the degree of distortion of freedom may be searched by testing various transformations of the model.
[0077] Further referring to step 204, Figure 2B An exemplary method 250 for generating a field according to some embodiments is shown. For example, in some embodiments, these techniques can be used to convert a 3D point cloud into a dense field. A dense field includes data at each field value, as further discussed herein (e.g., a point cloud lists only locations where there are points, and a nearby location without a point indicates that there are no points). As described above, the data can contain vectors. The field unit (x, y, z) represents the location where the vector appears in the field. In some embodiments, the field value inside a three-dimensional object in the 3D data can be set to zero. Therefore, in some implementations, the field can provide easier access to information compared to the 3D data (e.g., compared to the point cloud). In some embodiments, the field can make the 3D data look more uniform, while other 3D data (e.g., point clouds) may highlight specific areas. As another example, in some embodiments, these techniques can be used to convert a dense 3D image into a dense field. For example, the system can determine the gradient at each point of the dense 3D image and apply method 250, as further discussed herein.
[0078] In step 252, the machine vision system determines a set of vectors for each 3D data entry (e.g., 3D point). For example, the machine vision system can use neighboring 3D data point locations and / or information from a 3D sensor to determine surface normal vectors and edge vectors for each 3D data entry. The length of any vector can be zero (0), for example, to indicate that a particular data entry has no clear normals or edges.
[0079] In step 254, for each 3D data entry (e.g., point), the machine vision system determines the field cell (e.g., voxel) that contains each 3D data entry. In step 256, the machine vision system determines the cumulative data associated with each field cell. In some embodiments, these techniques include determining an input vector associated with each field cell. In some embodiments, the system can accumulate summary information about the vectors associated with all 3D data points located in the field cell. For example, the summary information can include the vector components themselves (e.g., when polarity is meaningful) and / or other information, such as the vector components vv generated by the outer product of each vector with itself. TThe matrix components formed (e.g., can be used when polarity is meaningless). In some embodiments, these techniques can include extending the range of influence of each point to blur or thicken features of the 3D data, etc. In some embodiments, the system can replicate in a predetermined pattern around each 3D data point. For example, the predetermined pattern can be relative to the direction of the vector. For example, these techniques can thicken the surface (e.g., by replicating the normal above and below), the edge (e.g., by replicating the edge in a cylinder around the crease vector), etc. The predetermined pattern may vary depending on what the vector represents, for example, whether the vector represents a normal or an edge.
[0080] In step 258, the machine vision system determines representative data for each field cell based on the accumulated data of step 256. In some embodiments, if a field cell is not associated with any accumulated data (e.g., not associated with any 3D data entry, e.g., no vector falls into the field cell), the field cell may be set to zero (e.g., when the field includes vectors, zero is used to refer to the zero vector).
[0081] In some embodiments, these techniques may include determining a representative vector for each field unit based on the accumulated data for that field unit determined in step 254. For example, the representative vector may be determined by extracting eigenvectors from the accumulated matrix (e.g., by taking the outer product of each vector vv with itself). T In some embodiments, a regularization constant may be added to the denominator, for example, to prevent division by zero, to reduce the length of the representative vector when there is less relevant data or similar situations. For example, by using the regularization constant k and the following equation, a matrix M may be calculated for a set of n vectors v, where the vectors v1 to v n (Although not shown in the equation for simplicity, for i=1-n, the sum is over all vectors v i (carried out on):
[0082]
[0083] In some embodiments, the machine vision system can store the matrix M when generating the field (e.g., for gesture testing). In some embodiments, representative data of the matrix M can be stored, for example, only the six unique values in the matrix, information representing the six unique values (e.g., only five of the six values (or less) because a constant multiple of the identity matrix can be added to make one of the values zero), etc.
[0084] In some embodiments, the machine vision system may determine the representative vector using the matrix M. For example, the machine vision system may determine the representative vector using eigendecomposition, as described above. The representative vector may be calculated using eigenvalues, eigenvectors, and / or both. In some embodiments, the eigenvalues may be used to determine the length of the representative vector. For example, the largest eigenvalue may be used to represent the maximum length. As another example, one or more other eigenvalues may be used in conjunction with the largest eigenvalue (e.g., to account for potential inconsistencies in vectors in the representative vector). In some embodiments, the eigenvectors may be used to determine the direction of the representative vector. For example, the eigenvector associated with the largest eigenvalue may be used to represent a primary direction. The eigenvalues may be multiplied by the eigenvectors to determine the representative vector. For example, the eigenvector associated with the largest eigenvalue may be multiplied by only the largest eigenvalue, by the difference between the second largest eigenvalue and the largest eigenvalue (e.g., when the largest and second largest eigenvalues have the same value, the value may be zero), and so on. In some embodiments, the eigenvectors may be multiplied by the square root of the eigenvalues, for example, to obtain the square root of the eigenvalue from vv T For example, the eigenvector associated with the largest eigenvalue can be multiplied by the square root of the largest eigenvalue, by the square root of the difference between the second largest eigenvalue and the largest eigenvalue, and so on.
[0085] In some embodiments, the field can be normalized. For example, the system can normalize the field by mapping the length of each vector (e.g., through a sigmoid function) without changing the direction. For example, normalization can be used to adjust the posture score response relative to a threshold. For example, some embodiments can simply classify the score results as passed (e.g., above a threshold) or failed (e.g., below a threshold). In some embodiments, the field can be normalized more stably (e.g., using a sigmoid function to output a length between 0 and 1).
[0086] In some embodiments, this field is not normalized. For example, in some embodiments, the raw data (e.g., vector length) may make sense without normalization. For example, the shorter the vector length, the less confidence / consistency there is in a normal or edge, while the longer the vector length, the higher the confidence. In some embodiments, scoring techniques (e.g., dot products as described herein) may include such data from a field (e.g., length), and therefore it may be desirable to use a non-normalized field.
[0087] In some embodiments, each field cell value can be based on an associated data entry in the 3D data, multiple data entries in the 3D data, and / or all data entries. In some embodiments, these techniques can include determining a vector for each field cell value. For example, the data entry for the three-dimensional data can include a list of points, and these techniques can determine the vector for each field cell based on the list of points. In some embodiments, as described above, these techniques can determine the value of a field inside an object in the 3D data. For example, a machine vision system can determine from a point cloud that one or more field cells are associated with an interior portion of an object and set the value to zero.
[0088] Referring to steps 206 to 212, as described above, method 200 can be used to perform a coarse stage of a 3D model alignment search in a 3D image. In some embodiments, method 200 can search for an approximate pose of the 3D model in the field, which pose can be further refined by subsequent steps. The approximate pose can include, for example, a 3D position including an (x, y, z) position and orientation data, such as roll, pitch, and / or yaw. Referring to step 206, in some embodiments, testing includes testing a set of probes of the 3D model against the field. For example, a machine vision system can test a set of probes of the model against the field to determine a score by adding the dot product of each probe and an associated vector in the field. In some embodiments, as combined Fig. 9 As further discussed, the score can be based on multiple values (e.g., multiple fields), such as crease edge information, occlusion boundary information, color information, intensity information, etc.
[0089] Figure 3 Exemplary three-dimensional image data 300 and associated fields 350 are shown according to some embodiments. Figure 2B The method 250 in the embodiment generates field 350. The 3D image data 300 includes a set of data entry items 302A to 302N (collectively referred to as data entry items 302). For each data entry item 302, the machine vision system can determine a set of vectors. For example, the machine vision system can determine vectors v1 to v2 for data entry item 302A. n ,like Figure 3 Field 350 includes a set of cells with associated values 352A through 352N (collectively referred to as cells 352). To convert 3D image data 300 into field 350, the machine vision system may perform one or more functions to convert the determined vectors into the associated field values. For example, as shown by value 352A, the machine vision system may perform one or more functions on vectors v1 through v2 of data entry 302A. nA function is executed on the data entry 302A to generate summary information "s" for the data entry 302A. As described herein, the result value s of the function can be a single vector, a scalar value (e.g., a vector length), and / or other summary information. The function can be executed for each data entry to populate the associated value of the field. Although Figure 3 Only a set of vectors v1 to v2 for data entry 302A is shown. n , but this is only for exemplary purposes to illustrate these techniques. As described herein, these techniques may include accumulating information about multiple data points used to determine the associated value of each field cell.
[0090] As described above, the summary information “s” can be converted into a final representative vector (not shown in Figure 3 ). For example, in some embodiments, the summary information "s" can be converted to a final representative vector by computing a component-by-component average that divides the summary information by the number of input items accumulated to the cell (e.g., optionally with a regularization constant in the denominator). As another example, in some embodiments, a feature vector can be extracted from a matrix used to accumulate information.
[0091] The inventors have determined that searching for a model in image data can be a time intensive process, whether the image data is 2D data or 3D data, because the process may require iteratively testing each pose of the model on the data. For example, when performing a search, there are dimensions of a search space (e.g., image data and / or fields such as 2D or 3D runtime data), and dimensions of a pose space (e.g., x, y, z, roll, pitch, yaw, scale, skew, orientation, perspective, and other non-linear distortions, etc.). The more the number of dimensions increases, the more poses there are to search for the model in the image data, which increases the processing required to search for the model.
[0092] The inventors have developed techniques for processing image data before searching for models. The processed image data allows the system to eliminate a large amount of potential gesture space during the search process. The machine vision system can be used to perform large spot checks in the search space and then refine those areas to significantly increase processing speed. As discussed further below, the processed image data allows the machine vision system to take arbitrarily large step sizes, so that it can still be ensured that the machine vision system will not miss any gestures, and when the gesture is refined, the gesture will obtain a good score (e.g., above a predetermined threshold). For example, this technique is contrasted with downsampling techniques, which can be used to increase search speed, but may miss gestures that would otherwise be considered gesture refinements. In some embodiments, the techniques discussed herein can also provide sampling of the same dimension (e.g., field) of the image data, thereby reducing a set of data input items of the image data to a single value. A set of values in the processed image data allows the machine vision system to test gestures to determine whether a model can be found at any associated gesture in the image data.
[0093] As further discussed herein, these techniques can generally divide the search into one or more layers. For example, these techniques can generate two different layers of image data, including a first layer of processed image data (e.g., layer 1) and a second layer of processed image data (e.g., layer 2) to create a larger search area of the image data, wherein each larger search area in the second layer cumulatively represents several smaller areas of the first layer. The system can process each larger area of the lower layer to determine whether it will search for gestures in smaller areas in the higher layer (e.g., where layer "1" is a higher layer of layer "2"). As described herein, the system can use a third layer and / or more layers, wherein each lower layer is generated using a larger area than the previous layer. In some embodiments, a larger area can be analyzed using a maximum score limit of an associated smaller area. The maximum score technique can be used to allow the system to determine that the score of the model will not be higher than the value in the area. Therefore, the system can use the maximum score technique to determine that the score of any possible gestures in the associated area in the higher layer is high enough to make these gestures worth further examination when testing the image data. Otherwise, the system can determine that the gesture is not worth further consideration in any data entry associated with the area.
[0094] Figure 4An exemplary method 400 of a coarse search according to some embodiments is shown. In step 402, a machine vision system receives image data of an object (e.g., 2D data or 3D data such as a point cloud, a dense 3D image, etc.). The image data includes a set of data entries, which may include gradient information, vectors, etc. In step 404, the system determines a set of regions of the image data. Each region includes a set of associated adjacent data entries from the image data. In step 406, the system generates processed image data using the set of regions. As further discussed herein, the processed image data includes a set of cells, each cell having an associated value. The system may generate the processed image data by determining, for each region, a maximum possible score for each data entry in a set of associated adjacent data entries from the image data. The system may set one or more of the processed image data values according to the determined maximum possible score. In step 408, the system tests the pose of the model using the processed image data to calculate a score for the pose. In step 410, the system determines whether the score meets a predetermined threshold. If the score does not meet the threshold, the method proceeds to step 412 and eliminates subsequent pose tests for one or more regions. If the score meets the threshold, the method proceeds to step 414 and stores the pose (eg, for subsequent refinement). In step 416, the system determines if there are more poses to test, if so, the method returns to step 408, otherwise the method ends at step 418.
[0095] Referring to step 402, the image data can be a vector field representing any type of data. For example, the vector field can represent one or more of (a) a surface normal vector, (b) a fold edge vector (e.g., for an actual edge of an object, such as the side of a book and / or an outline or occluded edge, such as the edge of a cup that is not the actual edge of a cup because the cup is cylindrical), and / or (c) a color edge vector, such as an edge based on the color of the object (e.g., where one color stops and another color begins, such as for a striped object). In some embodiments, multiple fields can be used to perform techniques, as further discussed herein. In some embodiments, the machine vision system can determine a vector field based on received image data (e.g., 2D and / or 3D image data).
[0096] Referring to step 404, the machine vision system may determine a set of regions in the image data. For example, each region may include the same number of data entries from the image data, as described below in conjunction with Figure 5-8For further discussion. For example, for 2D data, each region may include 16x16 data entry items, 16x8 data entry items, 8x8 data entry items, 4x4 data entry items, 4x3 data entry items, etc. As another example, for 3D data, each region may include 16x16x16 data entry items, 8x8x8 data entry items, 16x8x8 data entry items, 5x5x5 data entry items, etc.
[0097] In some embodiments, these regions overlap with other nearby regions. For example, Figure 5-8 As further discussed in , the machine vision system can iteratively move from data entry to data entry and determine regions based on adjacent data entry (e.g., a predetermined number of data entry at adjacent locations) such that there is some overlap between data entry in nearby regions. Figure 7-8 As further discussed, if the region is a 2x2x2 region of data entries in 3D data, then a first region may overlap with its adjacent region by four data units.
[0098] In some embodiments, configuring the machine vision system to determine regions where there is some overlap between adjacent regions can provide better posture testing. For example, having some overlap can better test postures than using non-overlapping regions. For example, assume that the machine vision system uses a model with multiple probes, and the probes can have different phases from each other. In this example, it is also assumed that the probes have integer intervals in terms of data units and / or regions, for example, so that the probes will fall within a specific data unit, rather than potentially falling at a location shared by multiple data units. In this example, if the machine vision system determines the region in a non-overlapping manner (e.g., so that the region does not share data entries with adjacent regions), the processed image data will reduce the resolution of the original image data, so the probe positions in the original image data may not be tested in the processed image data. As a simple illustrative example, assume that two adjacent probes of the model fall in side-by-side data entries of the original image data, and the machine vision system uses a 2x2 data entry region to determine the processed image data. In this example, one of the two probes will fall on a straight line between two regions in the processed image data at each measured posture (e.g., due to the reduced resolution of the image data when calculating the processed image data). Therefore, in this example, the machine vision system will not be able to test the probe against the maximum possible score within the 2x2 neighborhood in the processed image data. Because the machine vision system will not be able to test the probe in the processed image data, the machine vision system cannot correctly estimate the score of the model, for example, using the maximum possible score as described herein. Therefore, this can cause the machine vision system to incorrectly ignore poses that require further testing and / or test poses that do not include potential locations for the pose.
[0099] Calculating the area where some overlap is needed allows the machine vision system to test each pose. For example, if the maximum number of operations per data entry is calculated (e.g., combined with Figure 5-8 As described above, each possible probe of the machine vision system can directly hit the calculated value of the processed image data (e.g., so the machine vision system can test according to the correct 2x2 region of the original image data). The machine vision system can then advantageously move the model more than one cell (e.g., more than two cells) for the next test. The model can be moved in this manner because, due to the construction of the region and maximum score techniques discussed in this article, the machine vision system indirectly evaluates the complete 2x2 region in the higher-level image data through one test, so the next location that the machine vision needs to test is the next adjacent (non-overlapping) 2x2 region of the image data. Therefore, these techniques allow individual probes to skip re-evaluation of certain cells that have been evaluated in previous tests (implicitly), thereby enabling the system to advantageously move the model more than one cell.
[0100] Referring to step 406, these techniques can be used to use image data (e.g., a vector field as described above) to create processed image data (e.g., a new field) having the same or similar resolution, but where each value of the processed image data represents the maximum possible score of a model probe that falls within a set of image data entries that includes the particular data entry. For example, the scoring metric can have a maximum possible score that can be achieved for a particular probe and data entry of the image data (e.g., the score may be lower depending on the actual data for the probe). In order to allow the system to perform a conservative search for model poses in the processed image data, these techniques can determine the maximum possible score for the set of image data entries, assuming that the probe exactly matches each data entry, and use the maximum of these maximum scores as the resulting value of the processed image data. In this way, the probes can be tested against the resulting values to determine whether the data entries associated with these values are likely to have sufficient scores to make it worth testing separately for a set of independent data entries used to determine the resulting value.
[0101] In some embodiments, each probe in the model can be a unit vector, and a scoring metric can be used to score a particular pose of the model for the image data, the scoring metric comprising calculating a square dot product of the probe and the corresponding image data, the square dot product calculating the product of the length and cosine between them, so that the score is higher the more aligned. When using a square dot product, if the probes are perfectly matched, the score will be the square of the length of the data entry (e.g., a vector). Therefore, on a per data entry basis, the maximum score reflects the square of the length, which is the maximum score that can be achieved by any probe of unit length. In this example, each value of the processed image data is filled with the maximum value of the squared length of any data entry (e.g., a vector) found in the data entry region in the input image data. Therefore, these techniques can summarize the single data entry in the image data (or a smaller region) that has the maximum score for the entire region.
[0102] Referring to step 408, the processed image data can be tested as if it were image data used to test the model (e.g., derived vector fields such as 2D gradient fields and / or normal fields, edge fields, and color fields in 3D as explained above). For example, once the processed image data is determined, the model of a particular pose can be tested by testing the model's probes to determine a score (e.g., length sum). When testing the probes for a particular region, if the probes fall anywhere within the region, these techniques allow the system to determine that the model's score will not be higher than the value for that region because it is the maximum value for the entire region as described above. Therefore, since the score for each data entry associated with the region cannot be higher, the associated value for the region is an upper limit in the region.
[0103] Referring to steps 410-414, the postures tested can be more sparse while ensuring that the calculated score cannot be less than the actual score of the best posture in the range of postures tested. For example, as discussed further below, if the region is used for an 8x8 data entry group of 2D image data, then if the posture of the model does not meet the threshold of a particular value for the region, the system can skip testing any remaining values for the region at the next level, as described below in conjunction with Figure 6 30. Thus, if the system tests a set of probes for a model and calculates a score that does not meet a threshold, the system can remove all values associated with the tested region from further testing. If the pose is above the threshold, the system can make a note to review these associated data entries later for further refinement. Further refinement can include performing method 400 with a smaller region size and / or a refined search as discussed herein, etc.
[0104] Referring to step 418, the output of method 400 is a set of poses of the model that can exceed a given threshold. The range of poses for which the score cannot exceed the threshold does not need to be considered again. In some embodiments, as described above, method 400 can be applied in a pyramid scheme, where the output of each stage is the input of the next higher resolution stage. For example, the first stage can use a region of a specific size (e.g., sixteen data entries, eight data entries, etc.), the second stage can use a smaller region (e.g., twelve data entries, ten data entries, four data entries, etc.), and so on. Method 400 allows the machine vision system to perform a higher level search in the processed image data, which can allow the machine vision system to take a larger step size when testing a pose (e.g., moving multiple blocks in each dimension instead of one block), and substantially test multiple poses simultaneously, as discussed further below.
[0105] A machine vision system may execute method 400 to process various types of image data, including 2D and / or 3D data. Figure 5-6 Provides examples for processing 2D image data. Figure 7-8 Provides examples for processing 3D image data. Figure 5 , Figure 5 An exemplary portion of two-dimensional image data 500 and associated processed image data 550 is shown in accordance with some embodiments. The two-dimensional image data 500 includes a set of 2D data entries, 502A through 502N, collectively referred to as data entry entries 502 (e.g., cells of a grid) in this example. Each data entry 502 has an associated 2D vector, such as v1, v2, etc. (e.g., in this example, data entry 502A has vector v1, and data entry 502N has vector v4,). Figure 54, a system is used to process 2D image data using regions of four data entries in a 2x2 configuration. Two of the regions are shown as 504A and 504B. As shown by these exemplary regions, the system can be used to iteratively step through the 2D image to identify regions. In this example, as the system moves from each data entry (e.g., in an iterative manner from left to right, from top to bottom), each region includes the current data entry and the right data entry, the bottom data entry, and the data entry between the right data entry and the bottom data entry.
[0106] Continue to refer Figure 5 , processed image data 550 includes cells of values 552A to 552N, values 552A to 552N are collectively referred to as values 552. The machine vision system is used to execute a function that processes the data input items in each region to determine a result value of the associated cell in the processed image data. As described herein, the function can be a maximum score function using a vector of associated data input items in the region. For example, the system processes vectors v1, v2, v4, and v5 of associated data input items in region 504A using a function that calculates result value f1, and the system uses the result value f1 to determine the value of cell 552A. As another example, the system processes vectors v4, v5, v7, and v8 of associated data input items in region 504B using a function that calculates result value f7, and the system uses the result value f7 to determine the value of cell 552B.
[0107] Figure 6 A set of poses illustrating exemplary elimination of two-dimensional image data during a coarse search in accordance with some embodiments. Figure 6 Shows Figure 5 500 and associated processed image data 550 in FIG. As described herein, the score of the model is calculated based on multiple model probes. For exemplary purposes, Figure 6 It is shown that one of the model probes that landed on cell 552A was determined to have an insufficient score when testing the pose of the model. Since cell 552A was not considered at the layer shown in the processed image data 550, one layer was moved up to the two-dimensional image data 500, and thus the pose corresponding to the position of the probe being tested is no longer considered (e.g., for the refined search of the remaining candidate positions in the two-dimensional image data 500).
[0108] Therefore, if Figure 6 As shown, when testing at a layer represented by processed image data 550 (e.g., a layer where four data input items are summarized per value in the next layer 500), the image processing system can test across layers, and when testing only one gesture in layer 550, a group of gestures in the next layer 500 can be basically tested. Figure 5-6As shown in the example in , since the system tests a model (as described, where one of the many possible model probes falls at cell 552A), since f1 represents the maximum of the four data entries in region 504A (and also for other probes tested in accordance with the cell, not shown), the system can determine that since the pose score is insufficient, none of the corresponding poses in the two-dimensional data 500 can score high enough. Therefore, as shown, the high-level search performed in the processed image data 550 can allow the machine vision system to take larger steps when testing poses (e.g., moving two squares in each dimension instead of one square when using a 2x2 region, as shown in ). Figure 5 ), and essentially test multiple poses simultaneously (e.g., four poses at a time when a 2x2 area is used).
[0109] Figure 7 3D image data 700 and associated processed image data 750 are shown according to some embodiments. The 3D image data 700 includes a set of 3D data entries 702A through 702N, collectively referred to as data entry items 702 (eg, 3D vector fields). Each data entry item 702 has an associated 3D vector, shown as v 11 、v 12 Etc. In this example, each vector in the first layer is represented by V 1X where X indicates a particular data entry for that layer (e.g., data entry 702A has a vector v 11 , and data entry 702N has vector v 14 , in this example, because both are in the first layer). Figure 7 As shown, a system is used to process 3D image data using eight data entry areas, including four data entry items in one layer and four data entry items in the next layer. Two of the areas are shown as 704A and 704B, each area including four data entry items in the first layer and corresponding four data entry items in the second layer. As described herein, the system can be used to iteratively step through the 3D image to identify areas. In this example, as the system moves across each data entry item in the 3D image data 700 (e.g., by row, column, and depth), the system determines a new area that includes the current data entry item and the right data entry item, the bottom data entry item, the data entry item between the right data entry item and the bottom data entry item, and the corresponding four data entry items in the same position in the next layer.
[0110] Continue to refer Figure 7, processed image data 750 includes cells 752A to 752N having values, collectively referred to as cells 752. The machine vision system is configured to execute a function that processes the data input items in each region to determine result data of associated values in the processed image data. As described herein, the function can be a maximum score function using a vector of associated data input items in the region. For example, the system uses the calculated result value f 11 The function processes the vector v of associated data entries in region 704A. 11 、v 12 、v 14 、v 15 、v 21 、v 22 、v 24 and v 25 , and the system uses the resulting value f 11 To determine the value of cell 752A. As another example, the system uses the calculated value f 12 The function processes the vector v of associated data input items in area 704B 14 、v 15 、v 17 、v 18 、v 24 、v 25 、v 27 and v 28 , and the system uses the resulting value f 12 to determine the value of cell 552B.
[0111] Figure 8 A set of poses that exemplarily eliminate three-dimensional image data during a coarse search is shown in accordance with some embodiments. Figure 8 Shows Figure 7 700 and associated processed image data 750 in FIG. As described herein, the score of the model is calculated based on multiple model probes. For exemplary purposes, Figure 8 Only one of the model probes that lands on cell 752A is shown for illustrative purposes, but as described herein, each of the model probes is tested against the associated cell to determine if the pose score is insufficient. Since cell 752A is not considered at the layer shown in processed image data 750, one layer is moved to three-dimensional image data 700, and therefore the corresponding eight poses in region 704A are no longer considered (e.g., for a refinement search run on the remaining candidate locations in three-dimensional image data 700).
[0112] like Figure 7-8As shown, when applying the techniques to 3D image data, the pose space can become quite large due to the dimensionality. In some 3D images and / or 3D applications, only a small fraction of the large space of potential poses contains good candidate poses (e.g., because the object occupies only a small portion of the image). As another example, in some 3D images and / or 3D applications, small rotations far from the relevant area can begin to receive very low scores. These techniques can utilize high-level search (e.g., using Figure 7 The processed image data 750 shown in FIG. 7A ) can be used to quickly eliminate large groups of poses that do not achieve acceptable scores. A higher level search can allow the machine vision system to take larger steps when testing poses (e.g., Figure 7 As shown, when a 2x2x2 area is used, two blocks are moved in each dimension instead of one block), and multiple poses are tested essentially simultaneously (e.g., eight poses are tested at a time when a 2x2x2 area is used).
[0113] The inventors have determined that performing pattern matching using only a single field may not provide sufficient information. For example, some techniques use a single surface normal field to find probes on the surface of an object by trying various pose configurations and scoring each pose on the field. However, using only the surface normal field may be problematic, for example, when one or more large areas of the field scene have the same surface normals that are similar to the training object. For example, if a 3D model is trained on a book and there is a table in the field scene instead of the book, the book will have a large number of matches on the table when only the surface normal vectors are used. Therefore, if an initial rough search is performed using the normal field for the approximate location of the model, the information in the field may not be sufficient to ignore the initial pose in further refinement. The inventors have developed technical improvements to machine vision search techniques that use additional information in addition to a single field. As further discussed herein, these techniques may include using other information, such as information about crease edges, occlusion boundaries, color, intensity, etc. The additional information can be stored in one or more separate fields of the search process (e.g., so that the probe's normal data can be matched to the normal field, and the probe's edge data can be matched to the edge field). The machine vision system can test each type of data and use multiple tests to determine a final score for a particular pose (e.g., by aggregating a single score, etc.). By using multiple fields, other information about the object can be used to improve the effectiveness of the search process. For example, by searching for both normal and edge information, these techniques can eliminate poses that have a high score for the normal field but a low score for the edge field. As another example, these techniques can improve the system's ability to search for specific types of objects (e.g., uniform objects). For example, while it may be difficult to search for a specific pose of a can, the techniques may include more information about the can, such as color and / or reflectivity to improve the search (e.g., because the shape of the can alone may not be sufficient).
[0114] Fig. 9An exemplary method 900 for performing an initial search for a three-dimensional model pose according to some embodiments is shown. In step 902, a machine vision system stores a three-dimensional model including a set of probes. In step 904, the system receives 3D data (e.g., a point cloud, a depth image, etc.) of an object, the 3D data including a set of 3D data entries. In step 906, the system converts the three-dimensional data into a set of two or more fields. The first field includes a first set of values, each set of values representing a first feature of an associated data entry or multiple data entries of a 3D data entry. The second field includes a second set of values, each set of values representing a second feature of an associated data entry or multiple data entries of a 3D data entry. In step 908, the system tests the pose of the 3D model using a set of fields, including testing the set of probes against the set of fields to determine a score for the pose. In step 910, the system determines whether the score meets a predetermined threshold. If the score meets the threshold, the method proceeds to step 912 and stores the pose (e.g., for subsequent refinement). If the score does not meet the threshold, the method proceeds to step 914 and determines whether there are more gestures to be tested. If so, the method returns to step 908, otherwise the method ends at step 916.
[0115] Referring to step 902, the model can be a trained model, as described herein. Each probe can contain one or more vectors. For example, a probe can represent a located vector (e.g., (x, y, z) position and (x, y, z) direction). The probe can represent normal data, edge data, intensity data, intensity gradient data, and / or other information. Normal data may include points on a surface and their normal directions, etc. (e.g., normal probes). Edge data may include points on the edge of an object fold or an object crease and data along the direction of the fold or crease (e.g., edge probes). For example, intensity data may include information related to intensity, surface reflectivity, color, albedo, etc. For example, intensity data may reflect information related to a grayscale image and / or a color image (e.g., color on an object, label, etc.).
[0116] Referring to step 906, the first feature is different from the second feature. Therefore, step 906 generates at least two different fields for the 3D data. Similar to the model probe, the values in the fields can include various types of data, such as surface normal data (e.g., normal vectors orthogonal to the surface of the object), edge boundary data (e.g., edge vectors pointing to edges, creases, and / or other features in the image), intensity data, intensity gradient data, etc.
[0117] In some embodiments, these techniques include converting runtime 3D data into one or more dense 3D arrays called fields. In some embodiments, the system generates a 3D array for each field. The 3D array is indexed using three indices, one for each dimension. The system can be used to use the index of the 3D array to represent the x, y, and z positions of each value in the array. For example, the x, y, and z indices in the array can be x, y, z positions and / or converted to x, y, z positions using a transformation. Each value may include a vector of the same dimension (e.g., 1 or more). Each such vector can represent a point in or near an associated 3D data entry. The length of the vector can be zero (0), for example, when no point is found in or near the associated 3D data entry (e.g., when the data entry is inside an object in a 3D point cloud).
[0118] With reference to step 908 described herein, the test posture may include a probe trained according to each assumed posture conversion. In some embodiments, in order to determine the score of the posture, the system can calculate the sum of the dot products of each probe and the associated value in the field. The probe is tested according to a set of fields generated by the system. The system can be used to calculate a similarity index based on the probe scores of each field. For example, the system can average the scores of each field to determine the total score of the posture. As another example, the system is used to perform more complex operations to merge the individual scores of each field, such as linear weighting (e.g., a*score 1+b*score 2, etc.), nonlinear weighting (e.g., min (score 1, score 2)), etc.
[0119] These techniques can store gestures above a threshold for subsequent refinement. In some embodiments, the threshold is configured so that scores above the threshold represent local peaks in the score space (e.g., gesture space). For example, in some embodiments, in addition to checking whether a particular gesture meets a threshold, the system can also analyze the score of a particular gesture relative to the scores of neighboring gestures. The system can be used to store a subset of gestures, where the subset has a higher score than its neighbors in the gesture space.
[0120] The technology operated according to the principles described herein can be implemented in any suitable manner. The processing and decision blocks of the above flowcharts represent the steps and actions that may be included in the algorithms for executing these different processes. The algorithms derived from these processes can be implemented as software that is integrated with one or more single-purpose or multi-purpose processors and guides their operation, or can be implemented as functionally equivalent circuits, such as digital signal processing (Digital Signal Processing, referred to as DSP) circuits or application-specific integrated circuits (Application-Specific Integrated Circuit, referred to as ASIC), or can be implemented in any other suitable manner. It should be understood that the flowcharts included herein do not describe the syntax or operation of any specific circuit or any specific programming language or programming language type. On the contrary, the flowcharts show functional information that can be used by those skilled in the art to manufacture circuits or implement computer software algorithms to process specific devices that perform the technology types described herein. It should also be understood that, unless otherwise specified herein, the specific order of the steps and / or actions described in each flowchart only illustrates an implementable algorithm, and can be changed in the implementation and embodiments of the principles described herein.
[0121] Thus, in some embodiments, the techniques described herein may be embodied in computer executable instructions implemented as software, including application software, system software, firmware, middleware, embedded code, or any other suitable type of computer code. Such computer executable instructions may be written using any of a variety of suitable programming languages and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine.
[0122] When the techniques described herein are embodied as computer-executable instructions, these computer-executable instructions can be implemented in any suitable manner, including as a number of functional devices, each of which provides one or more operations to perform algorithmic operations according to these techniques. Although "functional device" is instantiated, it is a structural component of a computer system that enables one or more computers to perform specific operational roles when integrated with and executed by one or more computers. A functional device can be part of a software element or the entire software element. For example, a functional device can be implemented according to a process, or as a discrete process, or as any other suitable processing unit. If the techniques described herein are implemented as multiple functional devices, each functional device can be implemented in its own way without all being implemented in the same way. In addition, these functional devices can be executed in parallel and / or serially as appropriate, and can use shared memory on the computer on which it is executing, use a message passing protocol, or pass information between each other in any other suitable manner.
[0123] Typically, functional devices include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, the functions of functional devices can be combined or distributed in the system in which they operate as needed. In some embodiments, one or more functional devices that perform the technology herein can form a complete software package together. In alternative embodiments, these functional devices can be adapted to interact with other unrelated functional devices and / or processes to implement software program applications.
[0124] Some exemplary functional devices for performing one or more tasks are described herein. However, it should be understood that the functional devices and task divisions described are merely illustrative of the types of functional devices that can implement the exemplary techniques described herein, and that the embodiments are not limited to being implemented with any specific number, division, or type of functional devices. In some embodiments, all functions can be implemented in a single functional device. It should also be understood that in some embodiments, some functional devices described herein can be implemented with other devices, or implemented separately from other devices (i.e., as a single unit or separate unit), or a portion of these functional devices may not be implemented.
[0125] In some embodiments, computer executable instructions implementing the techniques described herein (when implemented as one or more functional devices or in any other manner) may be encoded on one or more computer-readable media to provide functionality to the media. Computer-readable media include: magnetic media, such as disk drives; optical media, such as Compact Disks (CDs) or Digital Versatile Disks (DVDs); persistent or non-persistent solid-state memory (such as flash memory, magnetic RAM, etc.); or any other suitable storage medium. Such computer-readable media may be implemented in any suitable manner. As used herein, "computer-readable media" (also known as "computer-readable storage media") refers to tangible storage media. Tangible storage media are non-temporary and have at least one physical structural component. In the "computer-readable medium" used herein, at least one physical property of at least one physical structural component may change in some way during the process of creating a medium using embedded information, the process of recording information thereon, or any other process of encoding a medium using information. For example, the magnetization state of a portion of the physical structure of a computer-readable medium may change during the recording process.
[0126] In addition, some of the above-mentioned techniques include actions to store information (e.g., data and / or instructions) in some manner for use by these techniques. In some embodiments of these techniques, such as embodiments in which these techniques are implemented as computer-executable instructions, the information can be encoded on a computer-readable storage medium. Where specific structures are described herein as advantageous formats for storing such information, these structures can be used to impart a physical organization to the information when encoded on a storage medium. These advantageous structures can then provide functionality to the storage medium by affecting the operation of one or more processors that interact with the information, for example, by improving the efficiency of the computer operations performed by the processors.
[0127] In some (but not all) embodiments, the techniques may be embodied as computer-executable instructions that may be executed on one or more suitable computing devices running in any suitable computer system, or one or more computing devices (or one or more processors of one or more computing devices) may be programmed to execute computer-executable instructions. The computing device or processor may be programmed to execute instructions when the instructions are stored in a manner accessible to the computing device or processor, such as stored on a data memory (e.g., an on-chip cache or instruction register, a computer-readable storage medium accessible via a bus, a computer-readable storage medium accessible via one or more networks and accessible by a device / processor, etc.). The functional device including these computer-executable instructions may be integrated with and direct the operation of: a single multi-purpose programmable digital computing device, a cooperative system of two or more multi-purpose computing devices that share processing power and jointly perform the techniques described herein, a single computing device or a cooperative system of computing devices (co-located or distributed in location) for performing the techniques described herein, one or more field programmable gate arrays (FPGAs) for performing the techniques described herein, or any other suitable system.
[0128] The computing device may include at least one processor, a network adapter, and a computer-readable storage medium. For example, the computing device may be a desktop or laptop computer, a personal digital assistant (PDA), a smart phone, a server, or any other suitable computing device. The network adapter may be any suitable hardware and / or software that enables the computing device to communicate wired and / or wirelessly with any other suitable computing device through any suitable computing network. The computing network may include wireless access points, switches, routers, gateways, and / or other network connection devices, as well as any one or more suitable wired and / or wireless communication media (including the Internet) for exchanging data between two or more computers. The computer-readable medium may be suitable for storing data to be processed by the processor and / or instructions to be executed by the processor. The processor is capable of processing data and executing instructions. Data and instructions may be stored on a computer-readable storage medium.
[0129] The computing device may additionally have one or more components and peripheral devices, including input and output devices. These devices may be used to present a user interface, etc. Examples of output devices that may be used to provide a user interface include a printer or display screen for visually presenting output and a speaker or other sound generating device for audibly presenting output. Examples of input devices that may be used for a user interface include a keyboard and a pointing device, such as a mouse, a touch pad, and a digital panel. As another example, a computing device may receive input information through speech recognition or in other audio formats.
[0130] Embodiments of implementing technology in circuits and / or computer executable instructions are described herein. It should be appreciated that some embodiments may exist in the form of methods, at least providing an example of the method. The actions performed as part of the method may be sequenced in any suitable manner. Therefore, when constructing an embodiment, the actions in the embodiment may be performed in a different order than described, which may include performing some actions simultaneously, even if the order of the actions is shown in the illustrative embodiment.
[0131] The various aspects of the above embodiments may be used alone, in combination, or in various arrangements not specifically discussed in the above embodiments, and therefore their application is not limited to the details and arrangements set forth in the above description or shown in the accompanying drawings. For example, the various aspects described in one embodiment may be combined in any manner with the various aspects described in other embodiments.
[0132] In the claims, the use of ordinal terms such as "first", "second", "third", etc. to modify claim elements does not itself mean any priority, preference or order of one claim element over another claim element, nor does it mean the temporal order of the actions of a method when they are performed, but is only used to distinguish a claim element with a specific name from another element with the same name (but using ordinal terms) in order to distinguish the claim elements.
[0133] Likewise, the phraseology and terminology used herein are for description and should not be considered limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein is meant to encompass the items listed thereafter and their equivalents as well as additional items.
[0134] The word "exemplary" is used herein to refer to an example, instance or illustration. Therefore, any embodiment, implementation, process, feature, etc. described herein as exemplary should be understood as an illustrative example and should not be understood as a preferred or advantageous example unless otherwise specified.
[0135] Thus, having described several aspects of at least one embodiment, it will be appreciated that various changes, modifications and improvements will readily occur to those skilled in the art. Such changes, modifications and improvements are considered part of this disclosure and are considered to be within the spirit and scope of the principles described herein. Therefore, the foregoing description and accompanying drawings are intended only as examples.
Claims
1. A computerized method for testing the pose of a three-dimensional model for three-dimensional data, the method include: storing a three-dimensional model, the three-dimensional model comprising a set of probes, each probe comprising data of an associated portion of the three-dimensional model; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; The three-dimensional data is converted into a set of fields, including: generating a first field containing a first set of values, wherein each value of the first set of values indicates a first characteristic of an associated one or more data entry items in the set of data entry items; and generating a second field containing a second set of values, wherein each value in the second set of values indicates a second characteristic of an associated one or more data entry items in the set of data entry items, wherein the second characteristic is different from the first characteristic; and Testing the three-dimensional model pose using the set of fields includes testing the set of probes against the set of fields to determine a score for the pose, wherein testing the set of probes against the set of fields includes: determining a first score for the set of probes tested according to a first set of values for the first field; determining a second score for the set of probes tested according to a second set of values for the second field; and A score for the gesture is determined based on the first score and the second score.
2. The method according to claim 1, in, Generating the first field and / or the second field comprises generating a three-dimensional array for each field, wherein: Each three-dimensional array includes a set of three indices including an index for each dimension; and Each three-dimensional array represents the x, y and z position of each associated value in the first set of values and / or the second set of values using the set of three indices.
3. The method according to claim 1, in, The set of probes, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, and / or intensity data.
4. The method according to claim 1, in, Testing the gesture to determine a score for the gesture includes summing a dot product of each probe in the set of probes and an associated value in the first set of values and / or the second set of values.
5. The method according to claim 1, further comprising: include: Testing multiple postures to determine multiple relevance scores; determining one or more of the plurality of gestures including a score above a predetermined threshold to generate a set of gestures; as well as The set of poses is stored for subsequent processing.
6. The method according to claim 5, in, Each pose in the set of poses represents a local peak of a plurality of associated scores, and the method further includes refining the set of poses to determine a top pose of the three-dimensional model.
7. A system for testing a 3D model pose on 3D data, the system comprising one or more processors, the one or more processors being configured to: storing a three-dimensional model, the three-dimensional model comprising a set of probes, each probe comprising data of an associated portion of the three-dimensional model; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; The three-dimensional data is converted into a set of fields, including: generating a first field containing a first set of values, wherein each value of the first set of values indicates a first characteristic of an associated one or more data entry items in the set of data entry items; and generating a second field containing a second set of values, wherein each value in the second set of values indicates a second characteristic of an associated one or more data entry items in the set of data entry items, wherein the second characteristic is different from the first characteristic; and Testing the three-dimensional model pose using the set of fields includes testing the set of probes against the set of fields to determine a score for the pose, wherein testing the set of probes against the set of fields includes: determining a first score for the set of probes tested according to a first set of values for the first field; determining a second score for the set of probes tested according to a second set of values for the second field; and A score for the gesture is determined based on the first score and the second score.
8. The system according to claim 7, in, Generating the first field and / or the second field comprises generating a three-dimensional array for each field, wherein: Each three-dimensional array includes a set of three indices including an index for each dimension; and Each three-dimensional array represents the x, y and z position of each associated value in the first set of values and / or the second set of values using the set of three indices.
9. The system according to claim 7, in, The set of probes, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, and / or intensity data.
10. The system according to claim 7, in, Testing the gesture to determine a score for the gesture includes summing a dot product of each probe in the set of probes and an associated value in the first set of values and / or the second set of values.
11. The system according to claim 7, in, The one or more processors are further configured to: Testing multiple postures to determine multiple relevance scores; determining one or more of the plurality of gestures including a score above a predetermined threshold to generate a set of gestures; as well as The set of poses is stored for subsequent processing.
12. The system according to claim 11, in, Each pose in the set of poses represents a local peak of a plurality of associated scores, and the one or more processors are further configured to refine the set of poses to determine a top pose of the three-dimensional model.
13. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the following actions: storing a three-dimensional model, the three-dimensional model comprising a set of probes, each probe comprising data of an associated portion of the three-dimensional model; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; The three-dimensional data is converted into a set of fields, including: generating a first field containing a first set of values, wherein each value of the first set of values indicates a first characteristic of an associated one or more data entry items in the set of data entry items; and generating a second field containing a second set of values, wherein each value in the second set of values indicates a second characteristic of an associated one or more data entry items in the set of data entry items, wherein the second characteristic is different from the first characteristic; and Testing the pose of the three-dimensional model using the set of fields includes testing the set of probes against the set of fields to determine a score for the pose, wherein testing the set of probes against the set of fields includes: determining a first score for the set of probes tested according to a first set of values for the first field; determining a second score for the set of probes tested according to a second set of values for the second field; and A score for the gesture is determined based on the first score and the second score.
14. The non-transitory computer-readable storage medium according to claim 13, in, Generating the first field and / or the second field comprises generating a three-dimensional array for each field, wherein: Each three-dimensional array includes a set of three indices including an index for each dimension; and Each three-dimensional array represents the x, y and z position of each associated value in the first set of values and / or the second set of values using the set of three indices.
15. The non-transitory computer-readable storage medium according to claim 13, in, The set of probes, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, and / or intensity data.
16. The non-transitory computer-readable storage medium of claim 13, in, Testing the gesture to determine a score for the gesture includes summing a dot product of each probe in the set of probes and an associated value in the first set of values and / or the second set of values.
17. The non-transitory computer-readable storage medium of claim 13, in, The instructions are further operable to cause the at least one computer hardware processor to: Testing multiple postures to determine multiple relevance scores; determining one or more of the plurality of gestures including a score above a predetermined threshold to generate a set of gestures; as well as The set of poses is stored for subsequent processing.
18. The non-transitory computer-readable storage medium of claim 17, in, Each pose in the set of poses represents a local peak of a plurality of associated scores, and the instructions are further operable to cause the at least one computer hardware processor to refine the set of poses to determine a top pose of the three-dimensional model.
19. A computerized method for testing the pose of a three-dimensional model for three-dimensional data, the method include: storing a three-dimensional model, the three-dimensional model comprising a set of probes, each probe comprising data of an associated portion of the three-dimensional model; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; The three-dimensional data is converted into a set of fields, including: generating a first field containing a first set of values, wherein each value of the first set of values indicates a first characteristic of an associated one or more data entry items in the set of data entry items; and generating a second field containing a second set of values, wherein each value in the second set of values indicates a second characteristic of an associated one or more data entry items in the set of data entry items, wherein the second characteristic is different from the first characteristic; and The three-dimensional model pose is tested using the set of fields to determine a score for the pose based on associated values of the set of probes and the first field, the second field, or both.
20. The method according to claim 19, in, Determining a score for the gesture based on the set of probes and associated values of the first field, the second field, or both includes: The dot product of each probe in the set of probes and a value associated with the first set of values of the first field, the second set of values of the second field, or both is summed.
21. The method according to claim 19, in, Generating the first field and / or the second field comprises generating a three-dimensional array for each field, wherein: Each three-dimensional array includes a set of three indices including an index for each dimension; and Each three-dimensional array represents the x, y and z position of each associated value in the first set of values and / or the second set of values using the set of three indices.
22. The method according to claim 19, in, The set of probes, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, and / or intensity data.
23. The method according to claim 19, further comprising: include: Testing multiple postures to determine multiple relevance scores; determining one or more of the plurality of gestures including a score above a predetermined threshold to generate a set of gestures; as well as The set of poses is stored for subsequent processing.
24. The method according to claim 23, in, Each pose in the set of poses represents a local peak of a plurality of associated scores, and the method further includes refining the set of poses to determine a top pose of the three-dimensional model.
25. The method according to claim 19, in, Testing the three-dimensional model pose using the set of fields to determine a score for the pose includes testing the set of probes against the set of fields.
26. The method according to claim 25, in, Testing the set of probes on the set of fields includes: determining a first score for the set of probes tested according to a first set of values for the first field; determining a second score for the set of probes tested according to a second set of values for the second field; and A score for the gesture is determined based on the first score and the second score.
27. A system for testing a three-dimensional model pose on three-dimensional data, the system comprising one or more processors, the one or more processors being configured to: storing a three-dimensional model, the three-dimensional model comprising a set of probes, each probe comprising data of an associated portion of the three-dimensional model; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; The three-dimensional data is converted into a set of fields, including: generating a first field containing a first set of values, wherein each value of the first set of values indicates a first characteristic of an associated one or more data entry items in the set of data entry items; and generating a second field containing a second set of values, wherein each value in the second set of values indicates a second characteristic of an associated one or more data entry items in the set of data entry items, wherein the second characteristic is different from the first characteristic; and The three-dimensional model pose is tested using the set of fields to determine a score for the pose based on associated values of the set of probes and the first field, the second field, or both.
28. The system according to claim 27, in, Determining a score for the gesture based on the set of probes and associated values of the first field, the second field, or both includes: The dot product of each probe in the set of probes and a value associated with the first set of values of the first field, the second set of values of the second field, or both is summed.
29. The system according to claim 27, in, Generating the first field and / or the second field comprises generating a three-dimensional array for each field, wherein: Each three-dimensional array includes a set of three indices including an index for each dimension; and Each three-dimensional array represents the x, y and z position of each associated value in the first set of values and / or the second set of values using the set of three indices.
30. The system according to claim 27, in, The set of probes, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, and / or intensity data.
31. The system according to claim 27, in, The one or more processors are further configured to: Testing multiple postures to determine multiple relevance scores; determining one or more of the plurality of gestures including a score above a predetermined threshold to generate a set of gestures; as well as The set of poses is stored for subsequent processing.
32. The system according to claim 31, in, Each pose in the set of poses represents a local peak of a plurality of associated scores, and the one or more processors are further configured to refine the set of poses to determine a top pose of the three-dimensional model.
33. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the following actions: storing a three-dimensional model, the three-dimensional model comprising a set of probes, each probe comprising data of an associated portion of the three-dimensional model; receiving three-dimensional data of an object, the three-dimensional data comprising a set of data input items; The three-dimensional data is converted into a set of fields, including: generating a first field containing a first set of values, wherein each value of the first set of values indicates a first characteristic of an associated one or more data entry items in the set of data entry items; and generating a second field containing a second set of values, wherein each value in the second set of values indicates a second characteristic of an associated one or more data entry items in the set of data entry items, wherein the second characteristic is different from the first characteristic; and The three-dimensional model pose is tested using the set of fields to determine a score for the pose based on associated values of the set of probes and the first field, the second field, or both.
34. The non-transitory computer readable storage medium of claim 33, in, Determining a score for the gesture based on the set of probes and associated values of the first field, the second field, or both includes: The dot product of each probe in the set of probes and a value associated with the first set of values of the first field, the second set of values of the second field, or both is summed.
35. The non-transitory computer readable storage medium of claim 33, in, Generating the first field and / or the second field comprises generating a three-dimensional array for each field, wherein: Each three-dimensional array includes a set of three indices including an index for each dimension; and Each three-dimensional array represents the x, y and z position of each associated value in the first set of values and / or the second set of values using the set of three indices.
36. The non-transitory computer readable storage medium of claim 33, in, The set of probes, the first set of values of the first field, and the second set of values of the second field include surface normal data, edge boundary data, and / or intensity data.
37. The non-transitory computer readable storage medium of claim 33, in, The instructions are further operable to cause the at least one computer hardware processor to: Testing multiple postures to determine multiple relevance scores; determining one or more of the plurality of gestures including a score above a predetermined threshold to generate a set of gestures; as well as The set of poses is stored for subsequent processing.
38. The non-transitory computer readable storage medium of claim 37, in, Each pose in the set of poses represents a local peak of a plurality of associated scores, and the instructions are further operable to cause the at least one computer hardware processor to refine the set of poses to determine a top pose of the three-dimensional model.
