Method and system for point cloud registration for image processing

Abstract

Methods and systems of point cloud registration are provided. The method features include obtaining image data of a plurality of images of a same scene from one or more cameras, the plurality of images including a first image and a second image each respectively associated with a first point cloud and a second point cloud, the first point cloud and the second point cloud each respectively forming the scene; generating a three-dimensional first overlap region and a second overlap region each respectively encompassing points from the respective first point cloud and second point cloud and having a position and a size in the first point cloud and the second point cloud; iteratively modifying the position or the size or both of the first overlap region and the second overlap region to generate a best so far first overlap region and a best so far second overlap region for each iteration until a convergence criterion is met; and establishing a transformation based at least in part on the best overlap regions, and the transformation including a rotation portion, a translation portion, or both to convert point positions between the first point cloud and the second point cloud.

Description

Background Technology

[0001] A camera array (e.g., set up on a tablet or smartphone) can be provided to capture multiple images of the same scene from different angles. The same effect can be achieved by moving a single camera to capture images of the same scene from different angles. These images can then be used to generate depth maps, which in turn generate 3D geometric or semantic models to accurately locate objects in the 3D space of the scene. This is done to be able to measure objects in the scene, or the distances between objects in the scene (or from the camera to the object), for applications such as computer vision, artificial intelligence, object recognition, head-mounted 3D displays (HMDs), or point-of-view displays when it is desired to know the size, position, or identity of objects in the scene.

[0002] Many such conventional systems generate a point cloud for each image and then match points from different images of the scene viewed from different angles. Once these points are matched, triangulation or other algorithms can be used to measure the depth from the camera to these points, as well as other measurements. This is typically performed using the Iterative Closest Point (ICP) method. However, these methods are often inaccurate for systems that compare images along a sequence of frames generated from a moving camera. When the movement from one camera position to the next is too large, the system cannot accurately determine the correct corresponding point pairs, resulting in a high error rate. Furthermore, such systems are too slow because each iteration typically involves computation for all points in the point cloud being registered, which can cause pauses when the user views the image or delays when performing automated operations based on the position and / or identity of 3D objects in the scene. Attached Figure Description

[0003] The materials described herein are illustrated in the accompanying drawings by way of example, not limitation. For simplicity and clarity, the elements illustrated in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Additionally, reference numerals are repeated between drawings where deemed appropriate to indicate corresponding or similar elements. In the accompanying drawings:

[0004] Figure 1A and Figure 1B It is a pair of images of a scene, which has objects formed by point clouds and captured by a moving camera;

[0005] Figure 2 This is a flowchart of a point cloud registration method based on at least one implementation disclosed in this paper;

[0006] Figure 3This is a flowchart of a method for point cloud registration in image processing according to at least one implementation disclosed herein;

[0007] Figure 4 This is a flowchart of a spindle extraction method based on at least one implementation disclosed in this paper;

[0008] Figures 5A-5G This is a detailed flowchart of a method for point cloud registration in image processing based on at least one implementation disclosed in this paper;

[0009] Figure 5H This is a schematic diagram illustrating the alternative selection of the principal axis on the overlapping region of a cube, based on at least one implementation disclosed herein;

[0010] Figure 6 It is an image of the reconstructed scene obtained by point cloud matching using conventional methods;

[0011] Figure 7 It is an image of the reconstructed scene obtained by point cloud matching using conventional methods;

[0012] Figure 8 It is an image of the reconstructed scene obtained by point cloud matching using conventional methods;

[0013] Figure 9 It is an image of the reconstructed scene obtained by point cloud matching using conventional methods;

[0014] Figure 10 It is an image of the reconstructed scene obtained by point cloud matching using conventional methods;

[0015] Figure 11 The image of the reconstructed scene is obtained by point cloud registration for image processing based on at least one of the implementations disclosed in this paper;

[0016] Figure 12 This is a schematic diagram of the example system;

[0017] Figure 13 This is a schematic diagram of another example system; and

[0018] Figure 14 Another example device is shown, arranged entirely according to at least some implementations of this disclosure. Detailed Implementation

[0019] One or more implementations will now be described with reference to the accompanying drawings. While specific configurations and arrangements are discussed, it should be understood that such implementation is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements may be employed without departing from the spirit and scope of the description. It will also be apparent to those skilled in the art that the techniques and / or arrangements described herein can be used in a variety of other systems and applications different from those described herein.

[0020] While the following description illustrates various implementations that can manifest in architectures such as System-on-a-Chip (SoC) architectures, the implementations of the technologies and / or arrangements described herein are not limited to specific architectures and / or computing systems, but can be implemented by any architecture and / or computing system for similar purposes. For example, the technologies and / or arrangements described herein can be implemented using various architectures employing, for example, multiple integrated circuit (IC) chips and / or packages, and / or various computing devices and / or consumer electronics (CE) devices such as imaging devices, digital cameras, smartphones, webcams, video game panels or game consoles, set-top boxes, tablet devices with single or multiple cameras, wearable devices (e.g., HMDs), robots, etc. Furthermore, while the following description may elaborate on many specific details, such as logical implementations, types and interrelationships of system components, logical partitioning / integration choices, etc., the claimed subject matter can be implemented without such specific details. In other cases, to avoid obscuring the material disclosed herein, some materials, such as control structures and complete software instruction sequences, may not be shown in detail. The materials disclosed herein can be implemented in hardware, firmware, software, or any combination thereof.

[0021] The material disclosed herein can also be implemented as instructions stored on a machine-readable medium or memory, which can be read and executed by one or more processors. A machine-readable medium can include any medium and / or mechanism for storing or transmitting information in a machine-readable form (e.g., a computing device). For example, a machine-readable medium can include read-only memory (ROM); random access memory (RAM); disk storage media; optical storage media; flash memory devices; electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others. In another form, a non-transitory article of art, such as a non-transitory computer-readable medium, can be used with any of the foregoing examples or other examples, except that it does not include the transient signal itself. It does include those elements other than the signal itself, such as RAM, which can temporarily store data in a “transitory” manner.

[0022] The use of phrases such as "an implementation," "an implementation method," or "an example implementation method" in the specification indicates that the described implementation may include specific features, structures, or characteristics, but not every implementation may necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when describing a specific feature, structure, or characteristic in connection with an implementation, it is assumed that implementing that feature, structure, or characteristic in connection with other implementations (whether explicitly described herein or not) is within the knowledge of someone skilled in the art.

[0023] Systems, artifacts, and methods for point cloud registration in image processing are provided.

[0024] As described above, point cloud registration can be used to match identical points on objects in multiple captured images from multiple cameras in a camera array or from a single camera moving around a scene being captured. Point cloud registration can then be used, for example, for correction. Correction is performed to transform images from multiple cameras and thus from multiple angles (or multiple images from a single camera) into a common plane to simplify the feature matching process. The result is that each of the multiple images can now be viewed from the same viewpoint or angle, allowing the images to be placed in the same common plane, making it easier to match points in the images using parallel matching lines. Regardless of whether correction is performed, depending on the application used, the captured images can then be used for 3D reconstruction to form 3D depth maps, models, or other 3D spaces, or for other specific 3D reconstruction algorithms, such as Simultaneous Localization and Mapping (SLAM), to name just one example. The captured images can also be used for object recognition, to identify, recognize, and / or measure objects in 3D space, and for many different applications, such as artificial intelligence (AI), robotics and computer vision (e.g., autonomous vehicles and residential service robots), and point-of-view (POV) systems (e.g., with virtual, augmented, and mixed reality HMDs), to name just a few.

[0025] The task of point cloud registration is to compute the transformation matrix by minimizing the alignment error between two point clouds. Iterative nearest point (ICP) is the most popular technique in point cloud registration; please refer to [link to relevant documentation]. PJ Besl"A method for registration of 3-D shapes," IEEE Trans. Pattern Analysis, Mach. Intell, Vol. 14, No. 2, pp. 239-25 (February 1992). This conventional method iteratively performs a two-step process: point-to-point correspondence across the point cloud, and minimization of the least-squares distance metric describing the misalignment. It assumes that, in most cases, Euclidean nearest point pairs can find the correct correspondence. To satisfy this assumption, a common approach is to limit the magnitude of sensor variation, such as slowly moving the camera during visualization mapping, so that the distance moved for the same point from one image to the next is small. However, in practice, this constraint is often violated. For example, when a user is holding a camera recording a scene, due to human physical limitations, the user often simply cannot hold the camera smoothly enough to allow for such image processing. This is especially true when a user is trying to record a moving object and intentionally moves the camera to try to match the object's motion (often moving the camera too fast for accurate 3D processing).

[0026] Known improvements to classic ICP can be categorized into two types. First, some solutions select a subset of feature pairs to eliminate incorrect matches. One technique uses normal vectors to obtain more accurate point pairs. See also... J. Serafin "NICP: Dense normal based point cloud registration" by et al., IROS, pp. 742-749 (2015). Furthermore, a robust point matching (RPM) algorithm is proposed, which uses soft assignment (…). softassign Instead of binary allocation, it allows for better point selection and matching. See also Gold "New algorithms for 2D and 3D point matching: Pose estimation and correspondence," Advances in Neural Information Processing Systems, pp. 957-964 (1995). Other recent methods related to feature pairs combine two strategies to eliminate matching errors: pre-filtering and overlap filtering. See also... S. Nobili"Overlap-based ICP Tuning for Robust Localization of a Humanoid Robot," ICRA (2017). In each filtering technique, a set of false matches is eliminated. Some techniques attach descriptors to points to eliminate ambiguity, such as utilizing the signature (SHOT) of the orientation histogram. See also... Serafin People and Y. Zhuo "Automatic registration of partially overlapping three-dimensional surfaces", Mechanical Automation and Control Engineering, Proceedings of the International Conference, pp. 299-302 (2010). Nobili People and Tombari "Unique signatures of histograms for local surface description," European Conference on Computer Vision, Springer, Berlin, Heidelberg, pp. 356-369 (2010). The main drawback of all these methods is that they remain susceptible to noise, large viewpoint changes, and occlusion. Furthermore, many of the complex descriptors mentioned above are time-consuming.

[0027] Second, another solution for improving ICP is to use various error metrics to enhance robustness. Some techniques use multiple error metrics, such as point-to-plane and plane-to-plane matching; please refer to the relevant sections for details. Grant "Point-to-plane registration of terrestrial laser scans," *Photogrammetry and Remote Sensing J. ISPRS*, Vol. 72, pp. 16-26 (2012); and A. Segal "Generalized ICP" by [Authors' Name], Proceedings of Robotics: Science and Systems (RSS) (2009). "Iterative Reweighted Least Squares (IRLS)" is proposed as another example, using various cost functions to provide robustness against noise. See also [link to relevant documentation]. P. Bergström"Robust registration of point sets using iteratively reweighted least squares", Computational Optimization and Applications, 58(3): 543-561 (2014). Another technique introduced is the trimmed ICP algorithm, which extends the application of ICP to partially overlapping point clouds. See [reference needed]. D. Chetverikov "Robust Euclidean alignment of 3D point sets: the trimmed iterative closest point algorithm" by [Authors' Name], Image and Visual Computing, 23(3): 299-309 (2005). These methods are robust to a certain extent for a small number of matching errors. However, these techniques still cannot handle images with very little overlap in a scene or large rotations between images in a scene. For example, when the overlap (between two images) is less than 50% or the rotation between two images is greater than 60°, these techniques still lead to inappropriate point cloud matching, resulting in many inaccuracies and errors.

[0028] The drawbacks of those previous solutions can be categorized into three main aspects. First, techniques using low-level feature representations are often very noisy in practical applications. For example, conventional point detectors used in these techniques are typically very sensitive to various factors (including lighting, viewpoint, occlusion, etc.), leading to many erroneous point matches. Second, conventional low-level feature matching is usually performed in a dense matching space and for each iteration in ICP. For example, in classic ICP (see the reference above)... Besl(as described in the literature by [authors' name], in each iteration, points in the source are set to correspond to points in the target, which is the main reason for the slow speed of ICP. This may include brute matching, where every point in one point cloud is compared with every one or more points in another point cloud. This is so time-consuming that point cloud registration can cause significant delays in other processing and may unnecessarily consume a percentage of processor processing power. This alone is detrimental to the operation of the computer, computing device, or camera performing point cloud registration. Third, the error metric for point cloud registration can be based on local descriptors, but local descriptors are also sensitive to large rotations and small overlaps. Therefore, although some improvements have been proposed, these improvements are not ideal when there are very large rotations or very small overlaps between images (e.g., overlap rates below about 50%). Point cloud registration is almost unusable when the overlap between point clouds is less than 40% or the rotation angle is greater than 60°. For more information on these imperfect improvements, please refer to the literature cited above. Grant people, Bergström people, and Chetverikov Literature by [names of authors].

[0029] To address these issues, efficient and accurate methods and systems for point cloud registration in image processing can include iteratively modifying the position or size, or both, of a 3D overlapping region that surrounds a portion of a point cloud forming a scene in two images. These iterations can be used to modify the position and size of the overlapping region until convergence is achieved. During iterations, the overlapping region is also adjusted to align with the principal axes detected in the point clouds of both images. The best overlapping region for each image so far can be used as the initial overlapping region in the next iteration. A transformation including rotational components (e.g., rotation matrices) and translational components (e.g., translation vectors) can then be established using the overlapping region formed during the convergence iterations. This avoids heavy demands on storing and modifying data (e.g., weights of many points between different iterations), thereby reducing the memory capacity, processing time, and power consumption of computing devices.

[0030] To explain in more detail, given two point clouds P and Q, the registration task is to estimate the rotation matrix from P to Q. R Translation vector T To achieve this, overlapping regions can be used. B And the shape used here is a cube, in which B = ( x, y, z, w, h, l ), where the variable ( x, y, z ) are the coordinates of the center point of the overlapping region, while ( w, h, l) represents the width, height, and length dimensions of the overlapping area. For simplicity, the dimensions can be set to... w = h = l Overlapping areas B P and B Q Let P and Q represent the overlap, respectively. The joint estimate of the transformation and the overlapping region is expressed by the following equation to maximize their joint posterior probability:

[0031] (1)

[0032] in These are observed values, and It can be rewritten as:

[0033] (2)

[0034] The first probability on the right side of the equation represents the posterior probability of the overlap, while the second probability represents the conditional probability of the transformation given that their overlap is known. Assume the prior probability distribution of the overlap... p ( B P , B Q ) and the prior probability distribution of the observed values p ( Z All are uniformly distributed. According to Bayesian theory, the above equation (2) can be rewritten as:

[0035] (3)

[0036] Therefore, for R , T , B P , B Q The estimation is broken down into two subproblems: (1) by maximizing To overlap B P and B Q The problem of estimation, and (2) by maximizing To give and Transformation under different circumstances R and TThe problem of estimation. These two subproblems are interconnected and can be solved through an iterative process, in which the conditional probability of the observations can be represented by matching the principal axes of the point cloud to the overlapping region, while the posterior probability of the transformation can be represented by selecting the optimal overlapping region (and its matched principal axis) for the iteration. Thus, performing the disclosed method makes the probability equation (3) above effectively executed. This will be explained in detail below.

[0037] To accomplish these tasks, we first use the Manhattan world hypothesis. The Manhattan world hypothesis is an assumption about scene statistics for both urban and indoor scenes. See also Flint The paper, "A dynamic programming approach to reconstructing building interiors," ECCV (2010), by [Authors' Name], assumes that such city or interior room scenes are built on a Cartesian mesh, in which surfaces in the world are aligned with one of three principal directions, where all planes are either parallel or perpendicular to each other. It has been observed that most interior and city scenes are designed on the Manhattan 3D mesh. The disclosed method utilizes geometric priors from the Manhattan scene to define the principal coordinate system (or axes) of the point cloud and introduces a registration method based on the principal coordinate axes and a 3D overlapping region (here, a cube) representing the two point clouds to be registered.

[0038] Based on this representation, overlapping regions can be estimated with high accuracy, thereby accurately removing irrelevant points and achieving robust point cloud registration of multiple images with minimal overlap of the same scene. To achieve this, the principal coordinate axes are aligned during transform estimation. This can be done very quickly due to the ability to efficiently utilize a small search space that does not rely on traditional low-level feature matching, and it also achieves robustness against large rotations between images.

[0039] In terms of effectiveness and efficiency, the disclosed methods and systems achieve significantly better results than existing techniques. The disclosed methods significantly outperform existing methods when the overlap between point clouds is less than about 40% or the rotation angle is greater than about 60°. This is especially true when the overlap is less than about 30% or the rotation angle is greater than about 90°. This improvement in accuracy and reduction in computational load by reducing the power and processor time consumed in performing these tasks improves the operation of computers, computing devices, or cameras, thus freeing up more capacity and time for computing devices to perform other tasks. Further details are provided below.

[0040] Now for reference Figure 1AEach image 100 providing point clouds 102 to be registered with each other can have a point cloud constrained by bounding boxes 106. The bounding boxes 106 can be unique for each point cloud and can be automatically computed by finding, for example, the minimum and maximum coordinates of points on each default axis (x, y, z). In one form, a bounding box is the minimum possible bounding box that encloses the set of points in the point cloud.

[0041] As mentioned, the main unit manipulated by this method is the overlapping region 104. This overlapping region is placed within the point cloud 102 of 100, and subsequently within the bounding box, at the locations where the two point clouds of the two compared images may overlap. The overlapping region can thus enclose at least some points within the point cloud. Therefore, the bounding box is larger than the overlapping region.

[0042] refer to Figure 1B The example scene 110 shown in the two images 112 and 114 is presented from two different perspectives, whether captured from a single camera moving between the images or from multiple cameras, each providing one of images 112 or 114. As shown, each image 112 and 115 has its own overlapping region 118 and 116, respectively. In one form, the overlapping region is a cube, specifically a cubic, where the edge lengths in all three directions are the same (width, height, and length of the overlapping region). However, it should be understood that the method of this paper can be applied to overlapping regions with shapes other than cubes or cubics. And as described below, the method of this paper also modifies the position and edge length of the overlapping region. In this paper, position refers to the center point of the overlapping region, and edge length is the edge of the corner forming the overlapping region (where the two planes of the overlapping region intersect). However, it should be understood that other alternatives may be used, such as corners or other desired locations within or above the overlapping area as the locations discussed herein, and the edge length (also referred to herein as edge size) is also referred to as edge dimension, but for illustrative purposes herein, any overlapping dimension that conveniently and accurately indicates the size of the overlapping area may be used instead.

[0043] As mentioned, point cloud registration can be achieved through principal axis alignment within or on overlapping regions. Such principal axis alignment is represented by a principal axis pair with two detected principal axes 120 and 122 (detected using points in the point cloud). In this paper, a principal axis refers to a Cartesian coordinate system with three axes (x, y, z), and may be referred to herein as a principal coordinate system or principal axis system. Therefore, the term "principal axis pair" or "candidate principal axis pair" discussed below refers to two matching principal axes, each having three axes (x, y, z), wherein the x-axis of one principal axis in a pair matches the x-axis of the other principal axis in that pair, and so on for each of the three axes. Principal axes 120 and 122 are shown here aligned with each other and with overlapping regions 116 and 118, respectively. Details of such alignment will be provided below.

[0044] refer to Figure 2 The disclosed method 200 provides an overview of methods and systems for point cloud registration for image processing. In the illustrated implementation, process 200 may include one or more uniformly numbered operations, functions, or actions 202 to 210. References may be made herein by way of non-limiting example. Figure 12 The example image capture and processing system 1200 and any relevant places to describe process 200.

[0045] Processing 200 may first include: acquiring two captured images 212 and 214 pairs 202, each image having point clouds to be registered with each other. This can be an automatic process in which the point clouds are registered with each other at intervals of every two frames of a video sequence, or at some other interval, whether the desired registration is performed once or multiple times per video sequence, and can be determined differently depending on the application. For example, in SLAM 3D model applications, registration is typically performed every two adjacent frames, but for other 3D scene reconstruction applications, images are registered less frequently. In one form, if the camera moves slowly, this process is applied every other frame for 3D object reconstruction. Point cloud registration can also be performed at each consecutive scene or according to some other content-related indicator. In other cases, point cloud registration of the two images can be performed once or a number of other desired times when the user selects a specific mode on the computing device or camera (e.g., a 3D-related mode), or when setting a desired quality level or other computer capability level (e.g., battery, processing speed, or memory capacity-related settings). It should also be understood that the image can be a single-lens image or a still camera image.

[0046] Subsequently, the principal axis extraction operation 204 includes: extracting three principal orthogonal directions and naming them principal axes. This is performed by detecting planes 216 using the Manhattan world hypothesis, so that all resulting planes are parallel or perpendicular. The normals 218 of these planes are used to pair the perpendicular planes to utilize each pair of perpendicular planes to form a principal axis.

[0047] Once the initial overlap between point clouds is determined to be defined by the initial overlapping regions (e.g., Figure 1B The overlapping regions 116 and 118 in the diagram represent the points cloud being registered. An iterative process can then be used to determine the precise or optimal overlapping regions that can be used to establish the final transformation. Details regarding the selection of the initial overlapping regions are provided below. The iterative process may include: an estimate of transformation 210, which is achieved by using the alignment of the principal axes 222 and 224 of the two overlapping regions (representing the two point clouds being registered), and which are shown here as being formed during principal axis alignment operation 208. The estimate of transformation 210 represents the posterior probability described above. Furthermore, when the overlap is still considered unknown or too coarse, an estimate of the overlap can be used, and this can be referred to as an overlap tuning operation 206, which represents the conditional probability of the observations described above and is used to determine the optimal overlapping region 226. The estimation of the overlap and the transformation are coupled problems, and an integrated framework is proposed to jointly optimize them.

[0048] refer to Figure 3 This document provides a process 300 for a method and system of point cloud registration for image processing. In the illustrated implementation, process 300 may include one or more uniformly numbered operations, functions, or actions 302 to 308. This document may be referenced through non-limiting examples. Figure 12 The example image capture processing system 1200 and any relevant places to describe process 300.

[0049] Process 300 may include "acquiring image data of multiple images of the same scene from one or more cameras, the multiple images including a first image and a second image respectively associated with a first point cloud and a second point cloud, the first point cloud and the second point cloud each individually forming the scene" 302. As mentioned above, such pairing of images with point clouds to be registered can be performed in each video sequence or in consecutive scenes, at intervals of every two images, or at any other desired time interval.

[0050] Subsequently, process 300 may further include "generating a first and a second three-dimensional overlapping region, each region enclosing points from a respective first and second point cloud, and having a position and size within the respective first and second point clouds" 304. This may include generating initial first and second overlapping regions, each having a center point or other position indicator relative to a specific point in its point cloud. This may also include setting the edge lengths of one or more of the two point clouds. In one form, the center points of both overlapping regions originate from the center point of the bounding box of the point cloud, and the edge length is a factor of the bounding box size.

[0051] Individually, the process may include forming a set of candidate principal axis pairs, including: determining at least one principal axis formed using a first point cloud that will be paired with a principal axis formed using a second point cloud to generate principal axis pairs. As mentioned, this can be achieved by first using the Manhattan world hypothesis to detect planes used to form the principal axes. A principal axis may be formed by two normals, which are respectively normals to the detected perpendicular plane, and a third axis is calculated using these two normals to form three axes for each principal axis. Pairing of two principal axes can be performed by comparing histograms of color or luminance image data along each axis of the principal axes considered for pairing.

[0052] Then, process 300 may include "iteratively modifying the position or size, or both, of the first and second overlapping regions to generate the best first and second overlapping regions so far for each iteration, until a convergence criterion is met" 306. Specifically, this operation may include: generating a set of overlapping region samples that have variations in position or size, or both, relative to initial first and second overlapping regions that are initial with respect to individual iterations. The overlapping region samples are distinguished from the initial overlapping regions by using Gaussian-based random sampling of the variations in position, size, or both of the initial first or second overlapping regions.

[0053] The best principal axis pair from the candidate principal axis pairs is then iteratively matched with individual overlapping region samples, representing the posterior probability mentioned above. Determining the best match may include: distinguishing individual overlapping region samples that differ from the first or second overlapping region, where the initial center point position is maintained but the edge length is modified to match the edge length of the overlapping region sample to be compared. Before performing the distinction, the overlapping region sample may be adjusted to align with the principal axis of one of the candidate principal axis pairs, and the other initial overlapping region (with a fixed center point and an edge length equal to the edge length of the overlapping region sample) may be adjusted to align with the other principal axis in the candidate principal axis pair. In one form, the distinction includes: determining the Hamming difference between the binary histogram point distributions in the adjusted overlapping region sample and the initial overlapping region. Among all candidate principal axis pairs, the candidate principal axis pair that produces the minimum Hamming distance associated with the overlapping region sample can be considered a match for that overlapping region sample. In this paper, the Hamming distance refers to the difference in interval values ​​at the same bin position. Therefore, if the overlapping regions have intervals from 1 to 5, one overlapping region has the value [1 0 0 1 0], and the other overlapping region has the interval value [01 0 1 0], then the absolute value result is (1 – 0) + (0 – 1) + (0 – 0) + (1 – 1) + (0 - 0) = 2.

[0054] Furthermore, in each iteration, the best overlapping region sample and the best matching principal axis pair among all overlapping region samples can depend on the point distribution differences of the following two items: (1) the overlapping region sample of one of the first and second point clouds, and (2) the overlapping region of the other of the first and second point clouds, and this represents the conditional probability mentioned above. In one form, the Hamming distance has been calculated for the previous operation, and the Hamming distances can be compared with each other to determine which overlapping region sample is associated with the minimum Hamming distance, and thus becomes the best overlapping region sample for this iteration. An update test can be performed to determine whether the best overlapping region sample is better than the sample generated by the previous iteration. If not, the iteration continues by using the best overlapping region of the current iteration as the initial overlapping region for the next iteration. If at least one of the two best overlapping region samples is better than the previous overlapping region sample, a convergence operation is performed. Convergence occurs when the position or size of the best overlapping region between iterations, or the difference between the two, meets the convergence criterion (e.g., a threshold). Upon reaching convergence, the iteration stops, and the transformation is built using the last best overlapping region sample.

[0055] Process 300 may include “establishing a transformation based at least in part on these optimal overlapping regions, and the transformation includes a rotation component, a translation component, or both, to convert point positions between a first point cloud and a second point cloud” 308. This establishes a transformation that can be used to convert point positions in one point cloud to point positions in another point cloud. As mentioned above, this point cloud registration can then be used to generate 3D maps, 3D models, etc.

[0056] refer to Figure 4 A process 400 is provided for generating a set of candidate spindle pairs, which will be used to explain the operations of the detailed process 500 described below. In the illustrated implementation, process 400 may include one or more operations, functions, or actions 402 to 414, numbered evenly. References may be made herein by way of non-limiting example. Figure 12 The example image capture processing system 1200 and any relevant places to describe process 400.

[0057] Process 400 may include plane 416 and normal extraction 402 from the two point clouds to be registered with each other. A vertical plane is used to form axis pairs 404, and principal coordinates 420 are extracted 406. These three operations are performed for each point cloud. The principal axes 420 from each point cloud are then matched or classified 412 to form candidate principal axis pairs. In one form, all possible pairs are retained as candidates for iteration. In another form, only a certain number of principal axis pairs with the minimum difference between the paired principal axes (the first K candidate principal axes) are used for iteration.

[0058] Referring to Figure 5, a process 500 for a method and system of point cloud registration for image processing is provided. In the illustrated implementation, process 500 may include one or more uniformly numbered operations, functions, or actions 502 to 550. As a non-limiting example, reference may be made herein to... Figure 9 The example image capture processing system 900 and any relevant places are used to describe the process 500.

[0059] Process 500 may include “determining whether to process the Manhattan space” 501. In one form, the user can initiate this process only if the scene to be captured and used for point cloud registration has a structure suitable for the Manhattan world assumption (e.g., an indoor room or urban scene with many parallel and / or vertical planes). An example is an indoor room with four walls, a floor, a ceiling, and many tables or desks. This can also be performed automatically by a computing device, for example, by detecting that the computing device is indoors due to GPS, lighting, or other sensed environmental characteristics, thereby automatically initiating the process.

[0060] Processing 500 may include "acquiring image data of multiple images of the same scene" 502, and this may include acquiring images as frames of a video sequence or a series of still photographs. The captured scene may be a static scene or may have moving objects, and the camera may remain stationary or may move. Despite relatively little overlap and / or large differences in perspective between the images, this process 500 can still achieve better quality and better functionality for the computing device, as described below.

[0061] Processing 500 may include “perform preprocessing” 504, and the preprocessing is at least sufficient to perform point cloud registration, for example, when raw image data is received from one or more camera sensors. Such preprocessing may include noise reduction, pixel linearization, shadow compensation, resolution reduction, Bayer demosaicing, bokeh removal, and / or 3A-related operations (including auto white balance (AWB), auto focus (AF), and / or auto exposure (AE)), etc.

[0062] Process 500 may include "obtaining an image pair, in which both images have points of the same scene to be matched with each other" 506. Process 500 can be performed on two point clouds. Therefore, process 500 can be applied to, for example, only two RGBD images, but it can also be applied at intervals of every two images in a video sequence, as described above. This can be an option provided to the user, or it can be an option to be automatically turned on or off, as described above.

[0063] Process 500 may include "generating a point cloud for each image in the image pair within the bounding box" 508. Specifically, many different algorithms exist for point cloud generation, and these algorithms are typically based on some feature detection, where the edges or boundaries of objects are detected by algorithms that use gradients of color and / or brightness in the image data. The point cloud generation technique used here for RGBD images can simply convert the coordinates of the point cloud into 3D points, and can follow the rule (X, Y)=((x-cx) Z / f, (y-cy) Z / f), where (X, Y, Z) are the 3D coordinates in the camera coordinate system, (x, y) are the image coordinates, and (cx, cy, f) are the camera intrinsic parameters. For RGB images, depth is estimated first, and then the point cloud is estimated. For example, the following literature discloses depth estimation using stereo images: K. Zhang"Cross-Scale Cost Aggregation for Stereo Matching [C] / / Computer Vision and Pattern Recognition", IEEE, 2014: pp. 1590-1597; the following literature discloses depth estimation using video sequences: K. Karsch "Depth Extraction from Video Using Non-parametric Sampling" by [Authors' Name], European Conference on Computer Vision, Springer, Berlin, Heidelberg, 2012: pp. 775-788; and the following literature discloses depth estimation using a single image: D. Eigen "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network" by [Authors' Name], Department of Computer Science, Kurant Institute, New York University, 2014: pp. 2366-2374. For the remainder of process 500, the point cloud used for registration is denoted by P and Q.

[0064] Also refer to process 400 ( Figure 4 The process 500 may include “extracting candidate rotationally aligned principal axis pairs” 510. This may first involve the operation “detecting planes in two point clouds” 512 (and operation 402 ( Figure 4 For this operation, all possible planes can first be detected, regardless of their orientation. This can be done using techniques such as the Hough transform or Random Sample Consensus (RANSAC), to name just a few examples. Then, for the “Use Manhattan Hypothesis Plane” 514, only planes perpendicular to at least one other plane are kept.

[0065] Procedure 500 may include "forming a pair of perpendicular axes on the intersecting planes" 516. If two planes are detected to be perpendicular, the normals of these two planes are held together as the first two of the three axes to form the principal axes. Here, the normals are the unit normals of each plane (with a length specified as 1). See operation 404 ( Figure 4 As shown in the diagram, this operation is repeated for each detected vertical plane pair, where for each pair of axes... v 1 with vertical axis v2 pair.

[0066] Process 500 may include "forming a third axis for each axis pair" 518, and specifically, to form principal axes (or principal coordinate systems) S. The principal axes may include three orthogonal axes selected from the normal vectors of the primary detected planes. The three principal axes can be represented as... ,in This conforms to the right-hand rule. Therefore, the principal axis can be defined as S = ,in Each of the vectors is a 3×1 column vector, making S a 3×3 matrix. Each axis (or vector) is formed in a certain form. The three values ​​of ) are values ​​indicating the normal direction that forms the axis, or in other words, each axis is derived from the plane equation ( ax + by + cz + d = The normal vector of 0) a , b , c ], although other forms can also be used.

[0067] For each pair of normal vectors (or axes), if they are perpendicular to each other, then choose them as mentioned above. and The candidates are obtained, and the third principal axis is obtained through the following method:

[0068] (4)

[0069] Here, "×" represents the cross product of two vectors. This operation is also shown in procedure 400 (see [link]). Figure 4 Operation 406 in the process. The generation of the principal axes of two point clouds P and Q can also be represented as in the process ( Figure 4 Operations at 408 and 410 are shown. S P = { , i = 1 to C P}and S Q = { , i = 1 to C Q}

[0070] Process 500 may include "selecting the top K candidate axis pairs" 520. Point clouds can have many different principal axes. Therefore, process 500 can generate all or many individual candidate principal axes for P and Q. To determine the optimal overlap region during iteration, principal axes from one point cloud are matched with principal axes from another point cloud to form candidate principal axis pairs. This is done to ultimately generate aligned principal axes (or aligned coordinate systems).

[0071] if( If a principal axis is the alignment of P and Q, then the corresponding points in P and Q will only have translational relationships within the created coordinate system. In other words, each principal axis establishes its own coordinate system, such as (x, y, z) coordinates in 3D space. Therefore, when the coordinates of points in P and Q are transformed into the alignment of principal axes... When indicating a new coordinate system, where and (Among them, here " (This is matrix multiplication), the corresponding points in P' and Q' differ only in translation from each other, but not in rotation, thus simplifying the determination of the transformation.

[0072] Therefore, process 500 may include “performing axis matching” 522, whereby a principal axis of one point cloud may be matched with another principal axis of another point cloud to form a set of candidate principal axis pairs to be used during iteration, and to determine aligned principal axis pairs and optimal overlap regions. Principal axis matching can be performed using a set of heuristic rules. In one form, process 500 may optionally include “using color histograms” 524. With this option, the image data of the axes are projected into a color histogram for each axis, and the principal axes that are compared and have the smallest difference are designated as candidate principal axis pairs. Besides color histograms, other methods may also be used for principal axis matching. Principal axis matching is shown as process 400 (… Figure 4 Operation 412 in )

[0073] In this example, there may be a threshold cutoff number for the candidate principal axis pairs used for analysis. The selected candidate set can be represented as... , where k = 1 to K candidate principal axis pairs, and this is also shown as process 400 ( Figure 4 Operation 414 in ) . By means of Figure 5HIn one example, since there are six faces (e.g., 502-H and 504-H) in the overlapping region 500-H of the cube, there are at most 24 candidates and thus 24 possible different orientations. This is because for a principal axis in the cube (including a principal axis 506-H at each corner 508-H), one axis (e.g., the x-axis) can extend in any of three different directions (therefore, for axis = 24, there are 8 corners × 3 directions). Therefore, the maximum value of K here is 24. Note that the position of the candidate axis above or in the point cloud or overlapping region is irrelevant; a principal axis placed at a corner of the overlapping region of the cube can still represent the same principal axis extending in the same direction.

[0074] Alternatively, process 500 could use all or many individual candidate principal axis pairs instead of limiting the set to only the first K, but such a process could unnecessarily increase the computational load of point cloud registration.

[0075] Then, process 500 proceeds iteratively. Each iteration has two levels. Typically, the first level is to estimate (or re-estimate) the overlapping area. B P In the first level, overlapping areas B Q and B P The initial (and the initial overlapping region for each iteration is denoted below) is initialized. and ). B Q The points are fixed in point cloud Q, with their center positions remaining unchanged during the first stage. N overlapping region samples are generated to represent candidate overlapping regions in point cloud P. B P Then they are generated as overlapping regions with the initial region. B P Changes in position, edge length, or both. B Q The edge length is determined to match the edge length of each overlapping region sample to be compared throughout the first level and still within a single iteration. The optimal overlapping region is then determined during the first level by finding the maximum observation probability. B P As described below.

[0076] In the second level, the optimal overlapping region derived from the point cloud P and determined in the first level will now be... B P The center point is fixed in the point cloud P, and then the drawing is performed. N A number of overlapping region samples are used to represent candidate overlapping regions in the point cloud Q. BQ Similar to the first level, the overlapping area... B P The iteration proceeds by finding the probability of the maximum observed value, leading to the selection of the optimal overlapping region. B Q These two stages involve both transformation estimation and overlap tuning, which are performed alternately until the overlap region no longer changes (convergence is achieved). If convergence is not achieved, the optimal overlap region from the two stages of this iteration is selected. B P and B Q Used as the initial overlapping region in the next iteration B P and B Q .

[0077] Now, in more detail, process 500 may include “obtaining the first and second overlapping regions of the initial selection” 526. This initial operation refers to obtaining any parameters that are manually set by the user or automatically set by point cloud registration, 3D units, or any other application. As an example, this could include setting the desired edge length as a fractional multiple of, for example, the bounding box length of the point cloud. If the user or program automatically wants to use this method in a video sequence, different decimal numbers, such as “0.2”, “0.3”, and “0.5”, can be entered as fractional multiples of the edge length to be set to the overlapping region. The best or most efficient result can then be selected. Otherwise, the input might be a setting such as the decimal number “0.5”, which, while not guaranteeing the best result, can still provide acceptable results. Many other parameters can also be set in this way.

[0078] Process 500 may include "initializing the first and second overlapping regions" B P , B Q 528. This may include setting the location of the overlapping region (here, the center point) and the edge length (and consequently, the edge location). Therefore, process 500 may include "determining the center point of the first and second 3D overlapping regions" 530. In one form, the overlapping region... and It is initialized so that its center point is located at the center of point cloud P and Q respectively (representing the center coordinates of the 3D bounding boxes of P and Q).

[0079] Process 500 may also include "determining the initial length and position of the edges of the first and second overlapping regions" 532. As in the example provided here, when the overlapping region is a cube, only one number is needed as input for all three dimensions (width, height, and length of the overlapping region), and it should be understood that the center point of the 3D cube overlapping region has coordinates from the edge of the overlapping region and to half the edge length of each edge in any elevation view of the overlapping region. As mentioned, the initial length of the edges of the overlapping region can be set by the user or automatically by the 3D unit or some other application. In one example, the bounding boxes of point clouds P and Q are obtained. Then, the longest edge of the two bounding boxes is determined (denoted as "a"). The initial edge length of the overlapping region can then be set to a / 2, so the example input could be the decimal number "0.5". The user or application can set the initial edge length based on experiments or prior knowledge of the point clouds. If the prior information is not known, the initial decimal number can be set to a default value, such as "0.5". A good initial edge length can speed up processing and help improve registration accuracy, but this number does not significantly affect the final result because it is updated with each iteration, regardless of the input edge length.

[0080] Process 500 may include setting an iteration counter to "set t = 0" 534 to begin the first iteration. Then, process 500 may include "re-estimating the first overlapping region". "536 marks the beginning of the first stage of iteration. In this example, the first stage is considered to continue until operation 589. In the first stage, samples of overlapping regions with modified positions, sizes, or both are generated. These samples will be used to determine the optimal overlapping region by performing the transformation estimation and overlap tuning described above and explained in more detail below. The generation of the overlapping region samples can be explained as follows."

[0081] Process 500 may include "generating N overlapping region samples for n = 0 to N samples as..." The variation”538. In one form, for n = 0 to 49 samples, N = 50, but N can be set to other desired numbers. This operation of process 500 may include “generating relative to” Individual samples with variations in edge size, center point location, or both. "According to this example, each overlapping region sample has variations in both position and edge length. However, it will be understood that other examples are possible, such as changing the edge lengths of 25 samples (or some other number) and changing the center positions of 25 samples (or some other number). For example, another option could include generating samples with varying edge lengths in some iterations such as odd-numbered iterations t = 1, 3, 5, ... and generating samples with varying center positions in, for example, even-numbered iterations t = 2, 4, 6, ... Many other variations can be considered."

[0082] Process 500 may include "generating samples using a Gaussian distribution-based random sampling algorithm". "542. Therefore, the overlapping region samples can be generated completely randomly or at certain intervals as a ratio of the center position and edge length of the bounding box described above. Instead, a simplified form of Sequential Importance Resampling (SIR) particle filtering is applied, and this involves using a more efficient technique by employing a Gaussian function as the sampling likelihood. Further details of particle filtering are provided below by explaining overlap tuning. In this context, a Gaussian function can be applied to obtain the overlapping region samples n."

[0083] For example, position and size (edge ​​length) are sampled independently. Here, the edge length (or edge size) of the overlapping region sample is used as the "edge length". s "and the initial overlapping area" The edge length is For example, to generate a sample from all possible " s Select “” s The Gaussian function or distribution of the edge length can be expressed as:

[0084] (5)

[0085] Where σ is the deviation (and The variance (or variance) can be predetermined experimentally and provided by the 3D unit or other applications. In this implementation, the variance of the position can be set to 0.25. The edge length, and the variance of the edge length can be set to 0.2. Edge length, but other values ​​can be used.

[0086] The Gaussian function is obtained by approaching "s". When it provides a higher "s" value, while when "s" is far away A lower "s" value is provided to indicate which "s" is more likely to be selected, thus establishing the mean as a Gaussian function or distribution of the edge lengths of the optimal overlapping region selected in the previous iteration. Optimal overlapping region The mean is favored because we assume that it is obtained in the t-th iteration. and The probability of a conditional observation is higher or equal to that in the (t-1)th iteration. Gaussian sampling can be implemented using many different techniques.

[0087] For a change in position (or center point), the position can be represented as a vector c = (x, y, z), which is the center point of the cube, and its Gaussian function is...

[0088] (6)

[0089] in yes The center point, and It is the covariance (identity) matrix:

[0090] (7)

[0091] Still in the first level, process 500 may include "generating a product with the corresponding..." Variations in edge dimensions with the same edge size and with Individual samples with the same fixed center point location "544. Here, 'fixed' refers to making each overlapping area..." The center point and the initial (or latest) Keep it the same. Therefore, in the first level, when generating overlapping region samples... At that time, each sample changes in both its center position and edge size. Then, when n overlapping region samples are generated in the first level... When generated Edge size and Same, but center position is the same as The two overlapping regions are identical, so that they can be used as inputs in equation (X) below, thereby enabling the principal axis to be aligned (or matched) with overlapping regions of the same size. In other words, comparing overlapping regions of different sizes, and consequently point distributions of different sizes, may result in lower accuracy.

[0092] In the second level, operation 540 includes (in the second level, only the center point is modified to generate individual samples). Specifically, in addition to using a Gaussian distribution to generate [the data] at the second level... Similar operations can be performed outside the overlapping region samples. Therefore, in the second level, the center point is modified using equation (6) above, except that each "P" is swapped with "Q". However, all overlapping region samples The edge lengths are all fixed to the latest best The edge lengths are the same, while The center point and The center point is irrelevant. Overlapping region samples Will be with the latest best (Its edge length and center point remain unchanged) are compared (544).

[0093] Returning to the first-level description, transformation estimation can be performed using principal axis or principal coordinate alignment. The overlapping region can be set to have cubic faces, each of which is parallel to one of the principal axes of the point cloud. Once the overlapping regions are set in point clouds P and Q respectively... B P and B Q Given the center point and edge length, the problem in transformation estimation is to choose the values ​​that maximize the probability. Minimal optimal transformation rotation part R Translation section T And as explained above, given their overlapping regions, this probability represents the conditional probability of the transformation.

[0094] Given points in the overlapping region B, a binary point distribution histogram can be used. To describe its observations Overlapping areas B Classified to M In a cubic interval, where, in this disclosure, the size of each interval can be set to approximately ,function Defined as a binary histogram, where the values ​​in each interval are... (m=1, 2,… The value is 1 if the number of points in the interval is greater than the threshold, and 0 otherwise. Then, two overlapping regions are matched. and The cost can be defined as:

[0095] (5)

[0096] Where the function d() is the Hamming distance. To select the aligned coordinate system pair for P and Q, we define the likelihood of measuring the alignment probability given the overlapping region:

[0097] = (6)

[0098] Furthermore, the aligned coordinate system pair can be selected in the following ways:

[0099] = (7)

[0100] The main shaft Belonging to k = 1 to K candidate principal axis pairs S PQ One of them. As mentioned, K can be restricted to 24 possible orientations on the cube. Due to the sparse space used for principal axis alignment, rather than every possible principal axis pair in the test point cloud, this method can be very fast.

[0101] To achieve the spindle alignment equation (or the matching of the spindle with the overlapping region samples), process 500 may include activating a sample counter to "set n = 0" 545 to initiate the process. Thereafter, process 500 may include "setting..." B P = "546, "Settings" B Q = "548, and "Set k = 0" 550.

[0102] Process 500 may then include "aligning with spindle k" and The adjusted overlapping area is determined as ( × )and( × ) 552. This refers to matrix multiplication, and the overlapping region samples are... and overlapping areas The coordinates of the points in the middle are adjusted to the main axis. and The coordinates are used to align the overlapping region samples and the overlapping region with their respective principal axes.

[0103] Process 500 may include “determining the differences between the adjusted overlapping regions” 554, and in one form, this includes “determining the Hamming distance d between the binary point distribution histograms of each adjusted overlapping region” 556 according to equation (7) above. It should be understood that other equations may be used to determine the Hamming distance or the different representative differences between overlapping regions aligned with the candidate principal axes.

[0104] Process 500 may include "if Hamming distance exp 1 / d If it is the maximum value so far, then for n... Save as "558", where candidate principal axis pairs are saved as principal axis pairs with the maximum value of alignment equation (7) for potential alignment. The first value is automatically saved when alignment equation (7) is first used in the iteration.

[0105] Process 500 proceeds to the query "k=K?" 560 to determine if all candidate spindles have been tested. If not, process 500 may include "set k = k + 1" 562, and the process loops back to operation 552 to test the next spindle pair. To perform sampling of overlapping regions Alignment and matching. Once all candidate spindle pairs K have been tested, process 500 can include "saving the last one". Set as the best alignment of the principal axes for n ( )”564, so that the principal axis is now aligned with the overlapping region sample. Align (or match).

[0106] Then, process 500 may include querying “n=N?” 566 to determine whether all overlapping region samples N have been tested. If not, process 500 may include “setting n=n + 1” 567 to obtain the next overlapping region sample. and their corresponding overlapping areas Then, utilize all candidate axes. Test the next overlapping region sample until all (or some less than the expected total) overlapping region samples are tested. Match or align with the spindle.

[0107] Once the principal axis alignment is completed for each overlapping region sample, process 500 performs an overlap tuning operation to determine a single optimal overlapping region for iteration, which represents the posterior probability, as explained above. The basis of overlap tuning is as follows.

[0108] The overlapping region can be estimated recursively using Bayesian filtering. At the Bayesian prediction level, the overlapping region in P and Q is predicted, while at the update level, the overlapping region is updated based on the observations.

[0109] predict:

[0110] (8)

[0111] renew:

[0112]

[0113] (9)

[0114] Where "t" is the iteration time. The above prediction and update steps are performed iteratively until convergence. This iterative process is theoretically attractive, but due to... and The search space for combinations of these is large, making them difficult to handle.

[0115] Alternatively, it has been found that particle filtering can achieve an exact approximation of the Bayesian filtering described above. Specifically, the disclosed method can be a simplified form of the Sequential Importance Resampling (SIR) particle filter. The particle filter represents the likelihood P(X) through random sampling. It calculates the weight of each sample and selects those samples with the highest weights. The particle filter can be applied to the current point cloud registration method in the following manner.

[0116] The first part of particle filtering is prediction, and the method disclosed here predicts the location and size of the overlapping region based on a Gaussian distribution:

[0117] (10)

[0118] When the overlapping region sample N is determined as described above (as described in operation 542 above), it is applied to the current cloud registration method.

[0119] Next, weighting is performed to calculate the importance and / or weight of each sample. This is done using the Hamming distance equations (24) and (25) shown below for selecting the best overlapping region samples for the iteration.

[0120] Finally, SIR particle filtering involves resampling based on the calculated weights to update the samples in the next iteration. The difference between this method and the basic SIR particle filter is that at this stage, the original SIR method resamples the samples based on the distribution of the weights. In contrast, the current-point registration method only reuses the best overlapping region sample with the highest weight. Therefore, the resampling in this method generates only one sample, or it can be said that this method generates N duplicate samples by copying the best sample from the previous iteration.

[0121] Therefore, the probability density function A set of particles can be used To express it. However, and The joint sampling space is still too large, and a common solution to reduce the sampling space is to perform sampling one by one. and Sampling is performed. The formula can be approximated as:

[0122] predict :

[0123] (11)

[0124] renew :

[0125] (12)

[0126] predict :

[0127] (13)

[0128] renew :

[0129] (14)

[0130] in and These are the proposed prediction functions from the current state to the next state, which are derived from... and The model is centered on a Gaussian distribution and has been applied by sampling N from the overlapping region, and and It is the probability density function modeled by the particle filter.

[0131] As mentioned, and The sampling is performed separately and alternately. In the first stage, the process selects the best sample from the t-th iteration. Fixed as Then to Sampling and updating are performed. Then, in the second level, the process updates the sample by selecting the best sample from the (t+1)th iteration. Fixed as and to Perform sampling and updating. and The probability density function of their particles is used and Update:

[0132] (15)

[0133] (16)

[0134] in It is the Dirac delta function, and This represents the importance of the nth sample. This is modeled as follows:

[0135] (17)

[0136] (18)

[0137] Then, and Those selected as having the highest probability of observation:

[0138] (19)

[0139] (20)

[0140] in and The transformed overlapping region from P to Q in the t-th iteration.

[0141] To simplify the implementation of the first level, the weight equation (17) can be simplified as follows:

[0142] (twenty one)

[0143] It simply places the weights in the terms used by this method, where This is the best overlapping region so far (whether it was initial in the first iteration or from previous iterations). Then, for each candidate... The weighting can be:

[0144] (twenty two)

[0145] It can be simplified to select the best sample with the largest weight (and thus the smallest Hamming distance).

[0146] (twenty three)

[0147] The best value in this iteration is selected using the following formula. :

[0148] (twenty four)

[0149] in In the t-th iteration The best sample is the corresponding principal axis pair, × is matrix multiplication, and d() is the Hamming distance defined in equation (2). In other words, the Hamming distance has been calculated by equation (7) when the principal axis is aligned with the overlapping region sample, and now, in the case of storing the Hamming distance, it is only necessary to obtain the relevant Hamming distance of the analyzed overlapping region from memory.

[0150] To perform the same or similar process to tune :

[0151] (25)

[0152] in In the t-th iteration The best sample These are the corresponding principal coordinate system pairs.

[0153] To achieve this operation, process 500 may include "setting n = 0" 568 to reset the overlapping region sample counter. Then, process 500 may include "determining the optimal overlapping region for t". and the best spindle pair ( )”569, and this means applying the above equation (24) for the first level and equation (25) for the second level, as mentioned, which represents the posterior probability of the cloud registration method and is performed by weighting using a simplified SIR particle filter.

[0154] This operation may include "obtaining the best aligned spindle pair for n ( The Hamming distance d”570 is obtained by, as mentioned, by comparing the principal axis with the overlapping region samples. The relevant Hamming distance d was calculated during alignment.

[0155] Once the current overlapping region sample for analysis is obtained... If the Hamming distance is d, then process 500 can include: "If the Hamming distance d so far is the smallest, then for t, and( Save as best option 571. Otherwise, the current overlapping region sample Discarded. If so, store the current overlapping region sample. As mentioned.

[0156] Subsequently, process 500 may include the query "n=N?"572 to determine whether all overlapping region samples have been analyzed, and to determine the optimal overlapping region sample. If not, process 500 proceeds to “Set n = n + 1” 573, and the process loops back to operation 569 to apply equation (25) to the next overlapping region sample. This is to determine if it is the best overlapping region sample so far. When the last overlapping region sample N is reached, process 500 can then include "taking the last..." , , Set as , , "574, thus setting the optimal overlapping area for the iteration and the corresponding aligned principal axis pairs."

[0157] Then, process 500 may include "calculating the rotation matrix" "575, where in a given candidate principal axis pair In this case,R and T The choice of is equivalent to the choice of the aligned principal coordinate pair. Therefore, when the cost equation (5) above is used again, the conditional probability of the transformation given the overlap can also be measured by the following equation:

[0158] (26)

[0159] Due to the observed values The translation invariance of the transformation means that optimization of the transformation in terms of the rotation matrix can be transformed into:

[0160] (27)

[0161] in and It is from the main axis set The rotation matrix is ​​composed of the principal axes extracted from the matrix, k=1,2,…K, where K = 24 as described above. This is achieved by... and Applying to P and Q, the two point clouds are in parallel coordinate systems. Therefore, the rotation matrix from P to Q can be obtained by the following equation:

[0162] (28)

[0163] The description is of the aligned main axis pair ( , When considering variables related to P and Q, the equation for the rotation matrix from P to Q becomes:

[0164] (29)

[0165] Where R is the rotation matrix from P to Q, It is the optimal rotation matrix of R, and where , and Both are 3 × 3 matrices. After determining the optimal overlapping region through iteration, applying this equation, the equation becomes:

[0166] (30)

[0167] Then, process 500 may include "calculating translation" "576. After estimating the rotation matrix, the translation vector can be estimated using the following formula:"

[0168] (31)

[0169] And it is applied to the specific process here:

[0170] (32)

[0171] in and They are and The mean or center point within the range, and respectively, a 3 × 1 vector providing coordinates [x, y, z]. Here, the mean refers to the average coordinates, where:

[0172] (32.5)

[0173] Where p is points within.

[0174] Subsequently, process 500 may include "applying a test for updating" 577. This operation tests whether the optimal overlap region in one of the point clouds, realigned using the optimal rotation matrix, is close to the optimal overlap region in the other cloud. The difference between these two is compared to the same difference using the final initial or optimal overlap region. Therefore, for the first level, process 500 may include "for estimating..." Determine whether (578) is

[0175] (33)

[0176] in It is an estimate The best overlapping region so far in the first level, while It is an estimate The latest and best in the second level .exist Use a hat instruct and They have the same central position.

[0177] Similarly, for the second level, process 500 may include "for estimation" Determine whether (579) is

[0178] (34)

[0179] To apply the same test to the optimal overlap region of the second level These tests involve applying matrix multiplication to adjust the overlapping regions using indicated rotation matrices. The point location histograms here are generated as the other histograms described above, and the Hamming distance between the two histograms is determined.

[0180] Whether at level one or level two, process 500 may include the query "Does the test satisfy?" 580 or 581. If the test is satisfied for level one, process 500 may include "Determine convergence difference" 582. For level one, this involves the operation "Determine..." The latest best central location and the latest The difference between the center locations”583, and “determining” The latest optimal edge length and the latest "The difference in edge lengths between the edge lengths"584. These two differences can be the following Euclidean distance, although other types of distances can also be used.

[0181] (35)

[0182] (36)

[0183] in( ) and( They are respectively and The center position, and They are and The edge length.

[0184] Because of these two convergence differences, process 500 may include "updating" "585, to update the best R and T for iteration t to the best R and T so far for all iterations. Then, process 500 may also include "setting" "586, to match" Update.

[0185] Then, the process proceeds to initiate the second level. Therefore, process 500 may include repeating operations 536 to 588 (by replacing P with Q in operations 536 to 586) to estimate... "590. Therefore, besides being an estimate..." In addition, these operations are repeated. These operations are the same, except that different equations are applied as described above. Then, the process loops back to operation 536 to begin the second stage of iteration t.

[0186] If the update test is not satisfied in the first level, process 500 skips the determination of convergence differences and proceeds to operation 586 to "set". = Then the loop returns to operations 590 and 536, which are the second level of iteration t.

[0187] Once the second stage is completed and the optimal overlap area is determined... And if the update test (580) is satisfied, then process 500 can determine the convergence difference 582, and can include "determine". The latest best central location and the latest The difference between the center positions of the centers is 587. This can also be the following Euclidean distance.

[0188] (37)

[0189] in( ) and( They are respectively The center position. For second-order convergence differences, differences in edge length (or other dimensions) are not required.

[0190] Then, similar to operation 585 at the first level, process 500 may also include "update". "588. Afterwards, process 500 may include 'Settings'." "589 updated" .

[0191] Once both stages of the iteration are complete, process 500 may include the query "Is at least one update test satisfied?" 591 to determine what values ​​are needed to determine if convergence has been reached (where the overlapping region no longer changes or changes very little between iterations). If neither the first nor the second stage satisfies the update test, process 500 proceeds to "Set t = t + 1" 592, and the process loops back to operation 536 to begin the next iteration t. The next query is then "Are both tests satisfied?" 593. If both the first and second stages of the iteration satisfy both update tests, process 500 may include a convergence test: "Are the center position difference and the edge length difference both satisfied the convergence criteria?" 594. This asks whether all three convergence differences are less than a threshold.

[0192] and and (38)

[0193] in and This is a predetermined threshold set through experimentation or other methods. This threshold test has factors from two levels: the overlapping region location and edge length from the first-level test, and the overlapping region location from the second-level test. If all three criteria are not met, process 500 may include "Set t = t + 1" 595, and the process loops back to operation 536 to begin the next iteration. If all three criteria are met, process 500 may include "Save". as well as as well as , As a transformation between two paired images, "596", and the cloud registration process ends. Other applications or programs (such as 3D depth mapping or 3D model reconstruction) can then use this transformation, as described above.

[0194] When only one update test is satisfied in the iteration, process 500 may include "tests for convergence" 597. This includes "in the estimation..." Timely update test: using from targeted The difference in the center position of the previous iteration t-1"598, or "in the estimation Timely update test: using targeted The difference in center position and edge length of the previous iteration t-1"599. In other words, instead of the newly calculated convergence difference (as in operation 582), the convergence difference from the last iteration is used in the convergence threshold comparison test equation (38). Then, process 500 continues to operation 594, applying the threshold in equation (38) to determine whether convergence exists, as described above.

[0195] It should be understood that other tests can be used instead of tests targeting the convergence threshold.

[0196] The methods described above can be summarized in the pseudocode style list as follows:

[0197] Input: Point clouds P and Q

[0198] Output: Rotation matrix Translation vector Overlapping cubes and The estimate.

[0199] By , Place it in the center of P and Q, and manually set their edge sizes to initialize them.

[0200] for t = 0 … T

[0201] estimate :

[0202] o Predicting based on Gaussian distribution The center point “c=(x,y,z)” and the edge size “s”. set up

[0203] o in yes It is the edge size and It was pre-ordered.

[0204] o in yes The center point and Set as identity matrix

[0205] o Generate 50 n=0,1,…49, each The central position and Same, and having the same n hour Edge size and same.

[0206] o for each cube pair ( ), n=0,…49 for principal coordinate alignment. The aligned coordinate system pair is represented as ( )

[0207] The input for this step is an overlapping cube. , And the output is the principal coordinate system pair.

[0208]

[0209] o Select the optimal pair of overlapping regions and their principal coordinate system (

[0210] o for each candidate Weighted:

[0211]

[0212] Therefore, the best sample with the highest weight is selected.

[0213] ■ (7)

[0214] Where d(.) is the Hamming distance defined in equation (2).

[0215] o , , ,

[0216] o Calculate cube pairs ( )of and

[0217] o ,

[0218] o ,

[0219] oUpdate , and

[0220] oIf

[0221] ■ , in( )and( They are respectively and The central position. They are and Edge size

[0222] ■ ,

[0223] o Resampling:

[0224] estimate :

[0225] o Predicting based on Gaussian distribution The center point is set to "c=(x, y, z)". n=0,1,2,…49.

[0226] o in yes The center and Set as identity matrix

[0227] o Set all Edge size and same.

[0228] o for each cube pair ( ), n=0,…,49 perform principal coordinate alignment. Represent the aligned coordinate system pair as ( )

[0229] (4)

[0230] o from all , k =0, 1, 2, ... 49 Select the optimal pair of overlapping regions and their principal coordinate system (

[0231] o for each Weighted:

[0232]

[0233] o Select the best sample with the highest weight

[0234] ■ (8)

[0235] ■

[0236] o is calculated based on equation (5-6) and

[0237] oUpdate , and

[0238] o

[0239] ■ in( )and( They are respectively and The center point.

[0240] ■ ,

[0241] o Resampling:

[0242] convergence

[0243] if and and If the iteration fails, the iteration stops; otherwise, it proceeds to the next iteration. and It is a threshold set manually by the user.

[0244] Finish

[0245] Evaluation

[0246] The publicly disclosed point cloud registration method was evaluated using two datasets. (Realsense) TM The dataset was captured to obtain a point cloud with 317 RGBD images. This dataset was tested for rotational variations in relation to point cloud registration. The second dataset is the 7Scenes dataset captured using Kinect®. See [link to relevant documentation]. J. Shotton"Scenecoordinate regression forests for camera relocalization in RGB-D images", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2930–2937 (2013).

[0247] refer to Figures 6-11 The images were reconstructed using five conventional methods as well as this cloud registration method, and then compared. The RGBD image pairs used show a scene with a kitchen and briefcase, but with very little overlap.

[0248] To test the performance of the registration algorithm, relative rotation error was used. and relative translation error The pairwise alignment error between the ground truth pose and the estimated pose was measured:

[0249] (39)

[0250] in It is a real rotation matrix. It is the estimated rotation matrix, and F Convert the rotation matrix into three Euler angles.

[0251] (40)

[0252] in It is the actual translation vector on the ground. It estimates the translation vector.

[0253] Robustness against strong rotations

[0254] First, regarding RealSense TM The performance of the point cloud registration method was tested using different rotations on the dataset. Different rotations were applied to the point cloud. The registration results are shown in Table 1 below.

[0255] Table 1. Registration results of point clouds with rotation

[0256]

[0257] (a) J. Yang et al. Go-ICP: An effective and globally optimal solution to the 3D registration problem, ICCVm, pp. 1457-1464 (2013).

[0258] It can be seen that when the rotation angle is greater than 60°, our method outperforms other traditional methods. This is because our method is based on principal axis (principal coordinate) alignment and principal axis extraction, which are theoretically invariant to rotation changes. Therefore, our method achieves relatively stable rotation estimation results under different rotation changes. In particular, when the rotation angle is greater than 90°, our method performs very well, while other methods fail completely.

[0259] Robustness for small overlaps

[0260] The registration performance of the point cloud registration method was also tested for point clouds with small overlap. (Source: 7scenes, see above reference) J. Shotton et al. The overlap range in the test image pairs of the dataset is 13% to 50%. The overlap rate between two point clouds P and Q ( or The intersection of these points is defined as the ratio of the number of points in their intersection to the number of points in their union.

[0261] (41)

[0262] Where T(P) is the point cloud transformed by the true transformation matrix. N() is the number of points. If the distance from a point in P to a point in Q is less than a threshold, it is considered as an element in the intersection of P and Q. The registration error is shown in Table 2 below. Examples of registration results are shown in... Figures 6-11 As shown in the figure, it can be seen that as the overlap rate decreases, this method provides consistently good results, while other conventional methods perform poorly.

[0263] Table 2. Registration error for smaller overlap

[0264]

[0265] refer to Figures 6-11 Image 600 shows the actual registered point cloud, image 700 shows the ICP result, image 800 shows the generalized-ICP result, image 900 shows the Go-ICP result, image 1000 shows the trimmed-ICP result, and image 1100 shows the result of this method.

[0266] Computational complexity evaluation

[0267] The computation time for each algorithm was counted to register two point clouds generated from VGA-sized RGBD image pairs. Each point cloud contained approximately 300,000 points, and the computational cost per iteration is shown in Table 3. Our method's computation time is only half that of ICP. Table 3 clearly demonstrates that our cloud registration method improves the capabilities of the computer, computing device, and / or camera required to operate on this cloud registration method.

[0268] Table 3. Computational costs of different methods

[0269]

[0270] in conclusion

[0271] The differences between the prior art and this disclosure are summarized in Table 4 below, and the advantages of this disclosure are listed in Table 5 below.

[0272] This disclosure proposes a novel framework for Manhattan point cloud registration. It utilizes geometric priors from the Manhattan scene, defines the principal coordinate system of the point cloud, and introduces a principal coordinate system-based registration method.

[0273] The disclosed method uses overlapping regions, and in one example, a cube is used for the overlapping representation, whereas traditional methods use low-level geometric descriptors such as points and lines. The advantage of this method is that it explicitly estimates the overlapping regions, which accurately removes irrelevant areas, resulting in robustness to smaller overlapping scenes.

[0274] The disclosed method estimates the transformation through principal coordinate alignment, while traditional methods rely on low-level and local feature descriptions and matching. Since principal coordinate alignment is based on global cube features, it is more robust to strong rotations than traditional methods.

[0275] The published method is fast because the master coordinates are spatially sparse, while traditional methods are slow due to the large number of feature correspondences performed in each iteration.

[0276] Table 4. Differences between the prior art and this disclosure

[0277]

[0278] Table 5. Advantages of this disclosure in terms of registration results

[0279]

[0280] also, Figure 3 and Figures 5A-5GAny one or more of the operations can be performed in response to instructions provided by one or more computer program products. Such program products may include signal-bearing media that provide instructions, which, when executed by, for example, a processor, provide the functionality described herein. Computer program products may be provided in one or more machine-readable media of any form. Thus, for example, a processor including one or more processor cores may perform one or more of the example processes described herein in response to program code and / or instructions or instruction sets delivered to the processor by one or more computer or machine-readable media. Generally, machine-readable media may deliver software in the form of program code and / or instructions or instruction sets that cause any device and / or system described herein to be executed as described herein. Machine or computer-readable media may be non-transitory articles of art or media (e.g., non-transitory computer-readable media) and may be used with any of the examples mentioned above or other examples, except that it does not include transient signals themselves. However, it does include those elements other than signals themselves, such as RAM, which may temporarily store data in a “transitory” manner.

[0281] As used in any implementation described herein, the term "module" refers to any combination of software logic, firmware logic, and / or hardware logic configured to provide the functionality described herein. Software may be implemented as a software package, code, and / or instruction set or instructions, and as used in any implementation described herein, "hardware" may, for example, individually or in any combination thereof, include hardwired circuitry, programmable circuitry, state machine circuitry, and / or firmware storing instructions executable by the programmable circuitry. Modules may be embodied collectively or individually as circuitry forming part of a larger system such as an integrated circuit (IC) or a system-on-a-chip (SoC). For example, a module may be embodied in logic circuitry to be implemented via software, firmware, or hardware of the coding system discussed herein.

[0282] As used in any implementation described herein, the term "logic unit" refers to any combination of firmware logic and / or hardware logic configured to provide the functionality described herein. A logic unit may be embodied collectively or individually as part of a larger system, such as an integrated circuit (IC), a system-on-a-chip (SoC), etc. For example, a logic unit may be embodied in logic circuitry to implement the firmware or hardware of the coded system discussed herein. Those skilled in the art will appreciate that operations performed by hardware and / or firmware may alternatively be performed via software, which may be embodied as a software package, code, and / or instruction set or instructions, and that a logic unit may also utilize a portion of software to implement its functionality.

[0283] As used in any implementation described herein, the term "component" may refer to a module or logical unit, as these terms are described above. Therefore, the term "component" may refer to any combination of software logic, firmware logic, and / or hardware logic configured to provide the functionality described herein. For example, those skilled in the art will appreciate that operations performed by hardware and / or firmware may alternatively be implemented via software modules, which may be embodied as software packages, code, and / or instruction sets, and it should also be appreciated that logical units may also utilize a portion of software to implement their functionality.

[0284] refer to Figure 12 The example image processing system 1200 is arranged according to at least some implementations of the present disclosure. In various implementations, the example image processing system 1200 may have an imaging device 1202 for forming or receiving captured image data. This can be implemented in a variety of ways. Thus, in one form, the image processing system 1200 may be one or more digital cameras or other image capture devices, and in this case, the imaging device 1202 may be camera hardware and camera sensor software, modules, or components 1204. In other examples, the image processing system 1200 may have an imaging device 1202, which includes or may be one or more cameras, and a logic module 1204 may be remotely or otherwise communicatively coupled to the imaging device 1202 for further processing of the image data.

[0285] Therefore, the image processing device 1200 can be one or more cameras on a multi-purpose device such as a smartphone, tablet, laptop, or other mobile device, or even a computer or other computing device. In other aspects, the device 1200 can be a tablet or other device with multiple cameras, wherein processing occurs at one of these cameras or at a separate processing location communicating with these cameras, whether the cameras are on or off the tablet or other device, and regardless of whether the processing is performed on a mobile device.

[0286] In any of these cases, such technology may include cameras such as digital camera systems, dedicated camera devices, or imaging phones or tablets, whether still image or video cameras, cameras with or without a preview screen, or some combination thereof. Thus, in one form, imaging device 1202 may include camera hardware and optics, including one or more sensors and controls for autofocus, zoom, aperture, ND filters, auto exposure, flash, and actuators. These controls may be part of a sensor module or component 1206 for operating the sensors. Sensor component 1206 may be part of imaging device 1202, or part of logic module 1204, or both. Such a sensor component can be used to generate images for a viewfinder and capture still images or videos. Imaging device 1202 may also have a lens, an image sensor with an RGB Bayer color filter, an analog amplifier, an A / D converter, other components for converting incident light into digital signals, and / or combinations thereof. In this document, digital signals may also be referred to as raw image data.

[0287] Other forms include imaging devices with camera sensors (e.g., webcams or webcam sensors or other complementary metal-oxide-semiconductor (CMOS) image sensors) that do not require a red-green-blue (RGB) depth camera and / or microphone array to locate the person speaking. The camera sensor may also support other types of electronic shutters (e.g., electronic shutters that are alternative to or replace rolling shutters and many other shutter types). In other examples, an RGB depth camera and / or microphone array may be used in addition to or as an alternative to a camera sensor. In some examples, imaging device 1202 may be equipped with an eye-tracking camera.

[0288] Imaging device 1202 may also include lens actuator 1208, which has a lens and lens driver or driver logic that receives a command to move the lens and applies current to move the lens. Imaging device 1202 may have any other relevant components for capturing images consistent with the methods and systems described herein.

[0289] In the example shown, logic module 1204 may include a preprocessing unit 1210 that receives and modifies the raw image data for further processing, as described above. The logic module may also include a 3D unit 1212 (which in turn includes a point cloud matching unit 1214) and optional depth map unit 1250, 3D model unit 1252, and / or other such applications that may use point cloud registration. In one form, depth map unit 1250 may be calibrated using point cloud registration transformation as mentioned above before reconstructing the 3D space.

[0290] Point cloud registration unit 1214 may have candidate principal axis pair unit 1216, initial 3D overlap region unit 1218, 3D overlap region sample unit 1240, candidate principal axis to sample matching unit (or principal axis (or coordinate system) alignment unit) 1242, optimal sample selection unit 1244, update unit 1246, and convergence unit 1248, wherein each unit is arranged to perform the operations mentioned above and to perform the tasks associated with the unit headings herein. Logic module 1204 may or may not be physically separated from the imaging device, and regardless of physical separation, logic module 1204 may still be considered part of the imaging device when performing tasks on data provided by the imaging device. It should also be understood that any of these units may include any code or hardware for performing the tasks suggested in the unit headings, and this code or hardware may or may not be physically separated from other objects or code performing other tasks, or may be an object or independent code segment.

[0291] Image processing system 1200 may have one or more processors 1220 (which may include a dedicated image signal processor (ISP) 1222 such as an Intel Atom), a memory bank 1224 that may or may not store point clouds or other data related to point cloud registration, one or more displays 1228 that provide image 1230, an encoder / decoder 1232, and an antenna 1234. In one example implementation, image processing system 100 may have a display 1228, at least one processor 1220 communicatively coupled to the display, and at least one memory 1224 communicatively coupled to the processor. Encoder / decoder 1232 and antenna 1234 may be provided to compress the modified image for transmission to other devices that may display or store the image. It will be understood that encoder / decoder 1232 may include a decoder for receiving and decoding image data for processing by system 1200, including point cloud registration performed by 3D unit 1214. Alternatively, the processed image 1230 may be displayed on display 1228 or stored in memory 1224. As shown in the figure, any of these components can communicate with each other and / or with multiple parts of the logic module 1204 and / or the imaging device 1202. Therefore, the processor 1220 can be communicatively coupled to both the imaging device 1202 and the logic module 1204 to operate these components. According to one method, although the image processing system 1200 (such as...) Figure 12 (As shown) may include a specific set of boxes or actions associated with a particular component or module, but these boxes or actions may be associated with components or modules that are different from the specific component or module shown here.

[0292] refer to Figure 13 The example system 1300 according to this disclosure operates one or more aspects of the image processing system described herein. It will be understood from the nature of the system components described below that such components may be associated with or used to operate a portion or portions of the image processing system 1800 described above. In various implementations, system 1300 may be a media system, but system 1300 is not limited to this context. For example, system 1300 may be incorporated into a digital still camera, a digital video camera, a mobile device with camera or video capabilities, such as an imaging phone, a webcam, a personal computer (PC), a laptop computer, an ultra-portable laptop computer, a tablet device with multiple cameras, a touchpad, a portable computer, a handheld computer, a PDA, a personal digital assistant (PDA), a cellular phone, a combined cellular phone / PDA, a television, a smart device (e.g., a smartphone, a smart tablet, or a smart TV), a mobile internet device (MID), a messaging device, a data communication device, and so on.

[0293] In various implementations, system 1300 includes a platform 1302 coupled to display 1320. Platform 1302 can receive content from content devices, such as one or more content serving devices 1330 or one or more content delivery devices 1340, or other similar content sources. A navigation controller 1350, including one or more navigation features, can be used to interact with, for example, platform 1302 and / or display 1320. Each of these components is described in more detail below.

[0294] In various implementations, platform 1302 may include any combination of chipset 1305, processor 1310, memory 1312, storage device 1314, graphics subsystem 1315, application 1316, and / or radio device 1318. Chipset 1305 provides communication between processor 1310, memory 1312, storage device 1314, graphics subsystem 1315, application 1316, and / or radio device 1318. For example, chipset 1305 may include a storage adapter (not shown) capable of providing communication with storage device 1314.

[0295] The processor 1310 may be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processor, an x86 instruction set compatible processor, a multi-core processor, or any other microprocessor or central processing unit (CPU). In various implementations, the processor 1310 may be one or more dual-core processors, one or more dual-core mobile processors, and so on.

[0296] The memory 1312 may be implemented as a volatile memory device, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM).

[0297] Storage device 1314 may be implemented as a non-volatile storage device, such as, but not limited to, a disk drive, optical disk drive, tape drive, internal storage device, attached storage device, flash memory, battery-backed SDRAM (synchronous DRAM), and / or network-accessible storage device. In various implementations, such as when including multiple hard disk drives, storage device 1314 may include techniques to add storage performance enhancement protection for valuable digital media.

[0298] The graphics subsystem 1315 performs image processing, such as still or video processing, for display. The graphics subsystem 1315 may be, for example, a graphics processing unit (GPU) or a visual processing unit (VPU). An analog or digital interface may be used to communicatively couple the graphics subsystem 1315 and the display 1320. For example, this interface may be any of a high-definition multimedia interface, a display port, wireless HDMI, and / or wireless HD-compliant technology. The graphics subsystem 1315 may be integrated into the processor 1310 or the chipset 1305. In some implementations, the graphics subsystem 1315 may be a standalone card communicatively coupled to the chipset 1305.

[0299] The graphics and / or video processing techniques described herein can be implemented in various hardware architectures. For example, graphics and / or video functions can be integrated within a chipset. Alternatively, discrete graphics and / or video processors can be used. As another implementation, graphics and / or video functions can be provided by general-purpose processors, including multi-core processors. In yet another implementation, these functions can be implemented in consumer electronic devices.

[0300] Radio device 1318 may include one or more radio devices capable of transmitting and receiving signals using a variety of suitable wireless communication technologies. Such technologies may involve communication across one or more wireless networks. Example wireless networks include (but are not limited to) wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular networks, and satellite networks. When communicating across such networks, radio device 1318 may operate according to any version of one or more applicable standards.

[0301] In various implementations, display 1320 may include any television-type monitor or display. Display 1320 may include, for example, a computer display screen, a touchscreen display, a video monitor, a television-like device, and / or a television set. Display 1320 may be digital and / or analog. In various implementations, display 1320 may be a holographic display. Additionally, display 1320 may be a transparent surface capable of receiving visual projections. Such projections may convey various forms of information, images, and / or objects. For example, such projections may be visual overlays for mobile augmented reality (MAR) applications. Under the control of one or more software applications 1316, platform 1302 may display user interface 1322 on display 1320.

[0302] In various implementations, one or more content service devices 1330 may be hosted by any national, international, and / or independent service and thus accessible via the Internet, for example, from platform 1302. One or more content service devices 930 may be coupled to platform 1302 and / or display 1320. Platform 1302 and / or one or more content service devices 1330 may be coupled to network 1360 to transmit (e.g., send and / or receive) media information to and from network 1360. One or more content delivery devices 1340 may also be coupled to platform 1302 and / or display 1320.

[0303] In various implementations, the content service device (one or more) 1330 may include a cable TV box, a personal computer, a network, a telephone, an internet-enabled device or appliance capable of delivering digital information and / or content, and any other similar device capable of transmitting content unidirectionally or bidirectionally between the content provider and platform 1302 and / or display 1320 via network 1360 or directly. It will be understood that content may be transmitted unidirectionally and / or bidirectionally to and from any component in system 1300 and the content provider via network 1360. Examples of content may include any media information, such as video, music, medical and gaming information, etc.

[0304] One or more content service devices 1330 may receive content, such as cable television programs, including media information, digital information, and / or other content. Examples of content providers may include any cable or satellite television, or radio or internet content provider. The examples provided are not intended to limit in any way the implementations according to this disclosure.

[0305] In various implementations, platform 1302 may receive control signals from navigation controller 1350, which has one or more navigation features. The navigation features of controller 1350 may be used, for example, to interact with user interface 1322. In some implementations, navigation controller 1350 may be a pointing device, which can be a computer hardware component (specifically a human-machine interface device) that allows users to input spatial (e.g., continuous and multidimensional) data into the computer. Many systems, such as graphical user interfaces (GUIs), televisions, and monitors, allow users to control a computer or television and provide data to it using physical gestures.

[0306] The movement of navigation features on controller 1350 can be replicated on the display (e.g., display 1320) by the movement of pointers, cursors, focus rings, or other visual indicators displayed on the display. For example, under the control of software application 1316, navigation features located on navigation controller 1350 can be mapped to virtual navigation features displayed on user interface 1322. In some implementations, controller 1350 may not be a separate component but may be integrated into platform 1302 and / or display 1320. However, this disclosure is not limited to the elements or scenarios shown or described herein.

[0307] In various implementations, for example, when enabled, the driver (not shown) may include technology that allows a user to instantly turn platform 1302 on and off like a television by touching a button after initial startup. Program logic may allow platform 1302 to stream content to a media adapter or(s) other content service device 1330 or(s) content delivery device 1340 even when the platform is “off”. Furthermore, chipset 1305 may include hardware and / or software support for, for example, 8.1 surround stereo audio and / or high-definition 7.1 surround stereo audio. The driver may include a graphics driver for an integrated graphics platform. In one implementation, the graphics driver may include a high-speed peripheral component interconnect (PCI) graphics card.

[0308] In various implementations, any one or more of the components shown in system 1300 may be integrated. For example, platform 1302 and(one or more) content serving devices 1330 may be integrated, or platform 1302 and(one or more) content delivery devices 1340 may be integrated, or platform 1302,(one or more) content serving devices 1330, and(one or more) content delivery devices 1340 may be integrated. In some implementations, platform 902 and display 1320 may be integrated units. For example, display 1320 and(one or more) content serving devices 1330 may be integrated, or display 1320 and(one or more) content delivery devices 1340 may be integrated. These examples are not intended to limit this disclosure.

[0309] In various implementations, system 1300 can be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, system 1300 may include components and interfaces suitable for communication via a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. Examples of wireless shared media may include portions of the wireless spectrum, such as the RF spectrum, etc. When implemented as a wired system, system 1300 may include components and interfaces suitable for communication via a wired communication medium, such as input / output (I / O) adapters, physical connectors connecting I / O adapters to corresponding wired communication media, network interface cards (NICs), disk controllers, video controllers, audio controllers, etc. Examples of wired communication media may include wires, cables, metal leads, printed circuit boards (PCBs), backplanes, switching structures, semiconductor materials, twisted pairs, coaxial cables, optical fibers, etc.

[0310] Platform 1302 may establish one or more logical or physical channels to transmit information. This information may include media information and control information. Media information may refer to any data representing content intended for a user. Examples of content may include data from voice conversations, video conferences, streaming video, email (“email”) messages, voicemail messages, alphanumeric symbols, graphics, images, videos, text, and so on. Data from voice conversations may include, for example, voice information, silence periods, background noise, comfort noise, tone, and so on. Control information may refer to any data representing commands, instructions, or control words intended for use in an automated system. For example, control information may be used to route media information through the system or instruct nodes to process media information in a predetermined manner. However, the implementation is not limited to this. Figure 13 The elements or situations shown or described in the text.

[0311] refer to Figure 14 This is an example of a small form factor device 1400 with various physical styles or form factors, in which system 1200 or 1300 can be embodied. In this way, device 1400 can be implemented as a mobile computing device with wireless capabilities. A mobile computing device can, for example, refer to any device having a processing system and a mobile power source or power supply (e.g., one or more batteries).

[0312] As described above, examples of mobile computing devices may include digital still cameras, digital video cameras, mobile devices with camera or video capabilities (such as imaging phones), webcams, personal computers (PCs), laptops, ultra-portable laptops, tablets, touchpads, portable computers, handheld computers, PDAs, personal digital assistants (PDAs), cellular phones, combined cellular phones / PDAs, televisions, smart devices (such as smartphones, smart tablets, or smart TVs), mobile internet devices (MIDs), messaging devices, data communication devices, and so on.

[0313] Examples of mobile computing devices may also include computers arranged for wearable by a person, such as wrist computers, finger computers, ring computers, glasses computers, belt buckle computers, armband computers, shoe computers, clothing computers, and other wearable computers. In various embodiments, for example, the mobile computing device may be implemented as a smartphone capable of performing computer applications in addition to voice and / or data communication. While some embodiments may be described using a mobile computing device implemented as a smartphone as examples, it will be understood that other embodiments may also be implemented using other wireless mobile computing devices. Implementations are not limited to this context.

[0314] like Figure 14 As shown, device 1400 may include a housing having a front 1401 and a rear 1402. Device 1400 includes a display 1404, an input / output (I / O) device 1406, and an integrated antenna 1408. Device 1400 may also include navigation features 1412. I / O device 1406 may include any suitable I / O device for inputting information into the mobile computing device. Examples of I / O device 1406 may include an alphanumeric keypad, numeric keypad, touchpad, input keys, buttons, switches, microphone, speaker, voice recognition device and software, etc. Information may also be input into device 1400 via microphone 1414 or may be digitized by a voice recognition device. As shown, device 1400 may include a camera 1405 (e.g., including at least one lens, aperture and imaging sensor) and a flash 1410 integrated into the rear 1402 (or elsewhere) of device 1400. Implementations are not limited to this context.

[0315] The various forms of devices and processes described herein can be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include: processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chipsets, microchips, chipsets, etc. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application programming interfaces (APIs), instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and / or software elements can vary depending on any number of factors, such as desired computational speed, power levels, thermal tolerance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed, and other design or performance constraints.

[0316] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium representing various logic within a processor, which, when read by a machine, cause the machine-manufacturing logic to perform the techniques described herein. Such a representation, referred to as an "IP core," may be stored on a tangible machine-readable medium and provided to various customers or manufacturing facilities for loading into the manufacturing machine that actually manufactures the logic or processor.

[0317] While certain features described herein have been referenced to various implementations, this description is not intended to be construed as limiting. Therefore, various modifications and other implementations of the implementations described herein that will be apparent to those skilled in the art to which this disclosure pertains are considered to fall within the spirit and scope of this disclosure.

[0318] The following examples involve alternative implementations.

[0319] According to one example implementation, a computer-implemented method for point registration in image processing includes: acquiring image data of multiple images of the same scene from one or more cameras, the multiple images including a first image and a second image, each of the first image and the second image being associated with a first point cloud and a second point cloud, which together form the scene; generating a three-dimensional first overlapping region and a second overlapping region, each of the first overlapping region and the second overlapping region enclosing points from the respective first point cloud and the second point cloud, and having a position and size in the respective first point cloud and the second point cloud; iteratively modifying the position or size, or both, of the first overlapping region and the second overlapping region to generate the best first overlapping region and the second overlapping region so far for each iteration, until a convergence criterion is met; and establishing a transformation based at least in part on the best overlapping region, the transformation including a rotation component, a translation component, or both, to transfer point positions between the first point cloud and the second point cloud.

[0320] According to another implementation, the method may include: determining convergence based on whether the position or size of the optimal overlapping region or the difference between the two in one iteration and another meets a convergence criterion; forming a set of candidate principal axis pairs, including: determining at least one principal axis formed by using the first point cloud, the at least one principal axis being paired with a principal axis formed by using the second point cloud; generating a set of overlapping region samples that change in position or size or both relative to an initial first overlapping region and an initial second overlapping region, the initial first overlapping region and the initial second overlapping region being initial with respect to an individual iteration; iteratively matching the optimal principal axis pair in the set of candidate principal axis pairs with an individual overlapping region sample; and in each iteration, selecting the optimal overlapping region sample and the best matched principal axis pair from all overlapping region samples based on the point distribution difference between (1) the overlapping region sample of one of the first point cloud and the second point cloud and (2) the overlapping region sample of the other of the first point cloud and the second point cloud.

[0321] The method further includes: pairing the principal axes of the first point cloud with the respective principal axes of the second point cloud; using the Manhattan world hypothesis to detect planes for forming principal axes; forming principal axes based on two normals that are respectively normals to the detected vertical planes, and using the two normals to calculate a third axis, to form three axes for each principal axis; for the principal axes considered for pairing, performing the pairing by comparing histograms of color or brightness image data along each axis, wherein each iteration has a first level and a second level, during the first level, the method includes: determining a plurality of first-level overlapping region samples as variations of the first overlapping region, and determining a plurality of second overlapping region variations, each of the second overlapping region variations retaining a center point of the second overlapping region, to compare the second overlapping region variations with individual first-level overlapping region samples to determine an optimal first overlapping region, and during the second level.

[0322] The method further includes: determining multiple second-level overlapping region samples as variations of the second overlapping region, and comparing the second-level overlapping region samples with the optimal first overlapping region to determine the optimal second overlapping region, wherein the variation of the first-level overlapping region samples is a variation relative to the first overlapping region in both the center point and edge size, and the variation of the second-level overlapping region samples is a variation relative to the second overlapping region in the center point, but the edge size used is the same as that of the first optimal overlapping region from the first level; in each iteration, determining a single optimal overlapping region corresponding to the first overlapping region and a single optimal overlapping region corresponding to the second overlapping region; and in the next iteration, combining these two single optimal overlapping regions in the following manner. Used as initial first and second overlapping regions: In the next iteration, the position and size of the optimal overlapping region are used in the first and second point clouds without maintaining other weighted data between iterations; given the observations of the first and second point clouds, the probabilities of the rotation and translation portions of the transformation, as well as the optimal first and second overlapping regions of the first and second point clouds, are estimated, wherein the estimation includes: determining conditional probabilities and posterior probabilities; and iteratively matching the optimal principal axis pair from the set of candidate principal axis pairs with individual overlapping region samples will be performed to represent the conditional probabilities, and selecting the optimal overlapping region sample and the matched optimal principal axis pair in each iteration will be performed to represent the posterior probabilities.

[0323] According to another implementation, a computer-implemented system for point cloud registration for image processing includes: a display; at least one processor communicatively coupled to the display; at least one memory communicatively coupled to the at least one processor; and a 3D unit operated by the processor through the following steps: acquiring image data of multiple images of the same scene from one or more cameras, the multiple images including a first image and a second image, each of the first image and the second image being associated with a first point cloud and a second point cloud, which together form the scene; generating a three-dimensional first overlapping region and a second overlapping region, each of the first overlapping region and the second overlapping region enclosing points from the respective first point cloud and the second point cloud, and having a position and size in the respective first point cloud and the second point cloud; iteratively modifying the position or size, or both, of the first overlapping region and the second overlapping region to generate the best first overlapping region and the second overlapping region so far for each iteration, until a convergence criterion is met; and establishing a transformation based at least in part on the best overlapping region, the transformation including a rotation component, a translation component, or both, to transfer point positions between the first point cloud and the second point cloud.

[0324] As an alternative, the system's 3D units can be provided to operate such that: the first overlapping region and the second overlapping region are initial overlapping regions for individual iterations, and wherein iteratively modifying the position or size, or both, of the first and second overlapping regions comprises: generating a set of overlapping region samples, which are variations of the first or second overlapping region, wherein the overlapping region samples are different from the initial overlapping region by using Gaussian-based random sampling of the variations in position, size, or both of the initial first or second overlapping region; generating a set of candidate principal axis pairs, each pair having a principal axis from the first point cloud and a principal axis from the second point cloud; and determining the best match between one of the candidate principal axes and the individual overlapping region samples, wherein determining the best match comprises: distinguishing individual overlapping region samples that are different from the first overlapping region, the individual overlapping region samples being different from one of a plurality of varied overlapping regions, the plurality of varied overlapping regions maintaining the initial center point position of the second overlapping region but varying to have edge lengths matching the edge lengths of the compared overlapping region samples, and determining the best match comprises: distinguishing individual overlapping region samples that are different from the second overlapping region, the individual overlapping region samples being different from the first overlapping region. The method further includes: adjusting the overlapping region sample to align with the principal axis of one of the candidate principal axis pairs, and adjusting the overlapping region by aligning the overlapping region to be compared with the overlapping region sample with the other principal axis of the candidate principal axis pair before performing the differentiation, wherein the differentiation includes: dividing the overlapping region sample and the overlapping region to be compared into intervals, and counting the number of points in the first point cloud and the second point cloud in each interval; the method includes: providing binary histograms of the overlapping region sample and the overlapping region to be compared by the following steps: providing 1 or 0 if the number of points in one of the intervals is greater than a threshold; providing the other of 1 or 0 if the number of points in one of the intervals is less than the threshold, wherein the difference that enables the differentiation is the Hamming difference between the histograms of the overlapping region sample and the overlapping region to be compared, and among all candidate principal axis pairs, the candidate principal axis pair that produces the minimum Hamming distance associated with the overlapping region sample is considered to match the overlapping region sample.

[0325] As another implementation, a computer-readable medium stores instructions that, when executed, cause a computing device to operate by: acquiring image data from one or more cameras of a plurality of images of the same scene, the plurality of images including a first image and a second image, each of the first image and the second image being associated with a first point cloud and a second point cloud, which together form the scene; generating a three-dimensional first overlapping region and a second overlapping region, each of the first overlapping region and the second overlapping region enclosing points from the respective first point cloud and the second point cloud, and having a position and size in the respective first point cloud and the second point cloud; iteratively modifying the position or size, or both, of the first overlapping region and the second overlapping region to generate the best first overlapping region and the second overlapping region so far for each iteration, until a convergence criterion is met; and establishing a transformation based at least in part on the best overlapping region, the transformation including a rotation component, a translation component, or both, to convert point positions between the first point cloud and the second point cloud.

[0326] As an alternative, these instructions are executed to cause the computing device to operate by: generating a set of candidate principal axis pairs, each pair having a principal axis from the first point cloud and a principal axis from the second point cloud; determining the best match between one of the candidate principal axes and each overlapping region sample, the overlapping region samples being variations of the first overlapping region or the second overlapping region in position or edge length or both; and determining the iterative best matching region by determining the overlapping region sample with the minimum difference among: (1) an overlapping region sample that is adjusted to align with the principal axis of the principal axis pair that matches the overlapping region and is associated with the first point cloud or the second point cloud, and (2) the other of the first overlapping region or the second overlapping region that is adjusted to align with the other principal axis of the matched principal axis pair, wherein the minimum difference refers to the difference in the point distribution associated with the adjusted overlapping region sample and the adjusted first or second overlapping region, wherein the minimum difference refers to the difference in the point distribution associated with the adjusted overlapping region sample and the adjusted first or second overlapping region, wherein Hamming distance has been established and used to match the principal axis with the overlapping region sample.

[0327] In another example, at least one machine-readable medium may include a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform the method according to any one of the above examples.

[0328] In yet another example, a device may include means for performing the method according to any one of the foregoing examples.

[0329] The examples above may include specific combinations of features. However, the examples above are not limited in this respect, and in various implementations, the examples above may include only a subset of such features, different orders of such features, different combinations of such features, and / or features other than those explicitly listed. For example, all features described with respect to any example method herein may be implemented with respect to any example device, example system, and / or example article, and vice versa.

Claims

1. A computer-implemented method, comprising: Image data of multiple images of the same scene obtained from one or more cameras, the multiple images including a first image and a second image, the first image and the second image being respectively associated with a first point cloud and a second point cloud, the first point cloud and the second point cloud respectively forming the scene; A three-dimensional first overlapping region and a second overlapping region are generated, each of which encloses points from the corresponding first point cloud and the corresponding second point cloud, and has a position and size in the corresponding first point cloud and the corresponding second point cloud. The positions and / or sizes of the first and second overlapping regions are iteratively modified to generate the best first and second overlapping regions so far for each iteration, until the convergence criterion is met. and The transformation is based at least in part on the optimal overlap region, and the transformation includes rotation and / or translation components to convert point positions between the first point cloud and the second point cloud. The method further includes: Forming a set of candidate principal axis pairs includes: determining at least one principal axis formed using the first point cloud, the at least one principal axis being paired with a principal axis formed using the second point cloud; Generate a set of overlapping region samples, which vary in position and / or size relative to an initial first overlapping region and an initial second overlapping region, which are initial with respect to an individual iteration; Iteratively match the best principal axis pair among the candidate principal axis pairs with individual overlapping region samples; and In each iteration, the best overlapping region sample and the best matching principal axis pair are selected from all overlapping region samples based on the point distribution difference between the overlapping region sample of one of the first point cloud and the second point cloud and the overlapping region sample of the other of the first point cloud and the second point cloud.

2. The method according to claim 1, comprising: Convergence is determined by whether the difference between the location and / or size of the optimal overlapping region in one iteration and another meets the convergence criterion.

3. The method according to claim 1, comprising: Pair the principal axes of the first point cloud with the principal axes of the second point cloud.

4. The method according to claim 3, comprising: The Manhattan world hypothesis is used to detect the planes that form the principal axis.

5. The method according to claim 4, comprising: The principal axes are formed by using two normals that are the normals to the detected vertical plane, and a third axis is calculated using the two normals, thus forming three axes for each principal axis.

6. The method according to claim 3, comprising: For the main axes that are considered for pairing, the pairing is performed by comparing histograms of color or brightness image data along each axis.

7. The method according to claim 1, wherein, Each iteration has a first level and a second level. During the first level, the method includes: determining a plurality of first-level overlapping region samples as variations of the first overlapping region, and determining a plurality of second overlapping region variations, each of the second overlapping region variations retaining a center point of the second overlapping region, to compare the second overlapping region variations with individual first-level overlapping region samples to determine an optimal first overlapping region, and During the second level, the method includes: determining a plurality of second-level overlapping region samples as variations of the second overlapping region, and comparing the second-level overlapping region samples with the optimal first overlapping region to determine the optimal second overlapping region.

8. The method according to claim 7, wherein, The variation of the first-level overlapping region sample is the variation in both the center point and edge size relative to the first overlapping region, and the variation of the second-level overlapping region sample is the variation in the center point relative to the second overlapping region, but the edge size used is the same as that of the optimal first overlapping region from the first level.

9. The method according to claim 1, comprising: In each iteration, a single optimal overlapping region corresponding to the first overlapping region and a single optimal overlapping region corresponding to the second overlapping region are determined. The two single optimal overlapping regions are used as the initial first overlapping region and the initial second overlapping region in the next iteration in the following manner: in the next iteration, the position and size of the optimal overlapping region are used in the first point cloud and the second point cloud without maintaining other weighted data between iterations.

10. The method according to claim 1, comprising: Given observations of the first and second point clouds, estimate the probabilities of the rotation and translation components of the transformation, as well as the optimal first and second overlap regions of the first and second point clouds, wherein the estimation includes: determining the conditional probability and the posterior probability; and Iteratively matching the best principal axis pair from the candidate principal axis pairs with individual overlapping region samples will be performed to represent the conditional probability, and In each iteration, the best overlapping region sample and the best matching principal axis pair will be selected to represent the posterior probability.

11. A computer-implemented system for point cloud registration in image processing, comprising: monitor; At least one processor communicatively coupled to the display; as well as Communication is coupled to at least one memory of the at least one processor; The at least one processor operates through the following steps: Image data of multiple images of the same scene obtained from one or more cameras, the multiple images including a first image and a second image, the first image and the second image being respectively associated with a first point cloud and a second point cloud, the first point cloud and the second point cloud respectively forming the scene; A three-dimensional first overlapping region and a second overlapping region are generated, each of which encloses points from the corresponding first point cloud and the corresponding second point cloud, and has a position and size in the corresponding first point cloud and the corresponding second point cloud. The positions and / or sizes of the first and second overlapping regions are iteratively modified to generate the best first and second overlapping regions so far for each iteration, until the convergence criterion is met. and The transformation is based at least in part on the optimal overlap region, and the transformation includes rotation and / or translation components to convert point positions between the first point cloud and the second point cloud. Wherein, the first overlapping region and the second overlapping region are the initial overlapping regions for individual iterations, and wherein iteratively modifying the position and / or size of the first overlapping region and the second overlapping region includes: generating a set of overlapping region samples, which are variations of the first overlapping region or the second overlapping region; and The at least one processor operates through the following steps: Generate a set of candidate principal axis pairs, each pair having a principal axis from the first point cloud and a principal axis from the second point cloud; and Determine the best match between one of the candidate principal axes and the individual overlapping region sample.

12. The system according to claim 11, wherein, The overlapping region samples are made different from the initial overlapping region by using random sampling based on a Gaussian distribution of the changes in position and / or size of the initial first overlapping region or the initial second overlapping region.

13. The system according to claim 11, wherein, Determining the best match includes: distinguishing individual overlapping region samples that are different from the first overlapping region, the individual overlapping region samples being different from one of a plurality of variable overlapping regions, the plurality of variable overlapping regions maintaining the initial center point position of the second overlapping region, but changing to have an edge length that matches the edge length of the compared overlapping region sample.

14. The system according to claim 11, wherein, Determining the best match includes: distinguishing individual overlapping region samples that are different from the second overlapping region, and the individual overlapping region samples are different from the first overlapping region.

15. The system according to claim 11, wherein, The at least one processor operates through the following steps: The overlapping region sample is adjusted to align with the principal axis of one of the candidate principal axis pairs, and the overlapping region is adjusted by aligning the overlapping region to be compared with the overlapping region sample with the other principal axis of the candidate principal axis pair before performing differentiation.

16. The system according to claim 15, wherein, The distinction includes: dividing the overlapping region sample and the overlapping region to be compared into intervals, and counting the number of points in the first point cloud and the second point cloud in each interval.

17. The system according to claim 16, wherein, The at least one processor operates by providing the overlapping region sample and the binary histogram of the overlapping region to be compared in such a way that: if the number of points in one of the intervals is greater than a threshold, then 1 or 0 is provided; if the number of points in one of the intervals is less than the threshold, then the other of 1 or 0 is provided.

18. The system according to claim 17, wherein, The difference that enables the distinction is the Hamming difference between the histograms of the overlapping region sample and the overlapping region to be compared, and among all candidate principal axis pairs, the candidate principal axis pair that produces the minimum Hamming distance associated with the overlapping region sample is considered to match the overlapping region sample.

19. At least one machine-readable medium storing a plurality of instructions, the instructions being responsive to execution on a computing device to cause the computing device to perform the method according to any one of claims 1 to 10.

20. An apparatus comprising means for performing the method according to any one of claims 1 to 10.

Images