Data processing apparatus and method for determining relative pose

Abstract

A data processing apparatus (100) and method for efficiently and accurately determining a relative pose (160) of a mobile agent such as a vehicle are disclosed. The apparatus (100) comprises processing circuitry (110) configured to determine a feature vector and a score for each point in a source point cloud (140) and a feature vector for each point in a target point cloud (150), and to transform a plurality of key points of the source point cloud (140) from a first pose to an unknown second pose for a plurality of relative pose candidates. The processing circuitry is further configured to pair each transformed key point with a target point in the target point cloud (150) based on distance, and to determine the relative pose of the mobile agent based on the plurality of feature vectors and positions of the plurality of transformed key points and target points.

Description

Technical Field

[0001] The present invention relates to a data processing apparatus and method for determining the relative pose of a mobile agent (e.g., a vehicle) between a first pose associated with a source point cloud and an unknown second pose associated with a target point cloud. Background Technology

[0002] LiDAR sensors can be used by mobile agents (such as vehicles) to mark distances to surrounding objects on a map and navigate their environment in real time. In this case, the LiDAR sensor collects multiple data points, also known as a point cloud. To localize a mobile agent using a LiDAR sensor, relative pose estimation between an online LiDAR scan (source point cloud) and a map-based LiDAR scan (target point cloud) may be required. These two point clouds may capture the same environment, such as the same segment of a road, but from different locations and directions. The challenge is to estimate the relative pose between these two point clouds so that if the source point cloud is transformed using the estimated relative pose, the two point clouds can be aligned. This allows the LiDAR sensor that captured the source point cloud to be accurately located in the coordinate system of the target point cloud. Summary of the Invention

[0003] The apparatus and method according to the invention help to efficiently and accurately determine the relative pose of a mobile agent (e.g., a vehicle) between a first pose associated with a source point cloud and an unknown second pose associated with a target point cloud.

[0004] This is achieved through the subject matter of the independent claims. Other implementations are apparent in the dependent claims, the specification, and the drawings.

[0005] According to a first aspect, a data processing apparatus is provided for determining the relative pose of a mobile agent between a first pose associated with a source point cloud and an unknown second pose associated with a target point cloud.

[0006] The data processing device includes a processing circuit, which is used to: determine the feature vector and score of each point in the source point cloud, and the feature vector of each point in the target point cloud.

[0007] Furthermore, the processing circuit is used to: select multiple key points in the source point cloud based on the score of each point in the source point cloud.

[0008] The data processing device is further configured to: for each of the multiple relative pose candidates, based on the corresponding relative pose candidate, transform the multiple key points in the source point cloud from the first pose to the unknown second pose, so as to obtain multiple transformed key points.

[0009] Furthermore, the processing circuit is used to: pair each transformed key point with a target point in the target point cloud to obtain multiple point pairs, wherein the distance from the corresponding target point to the corresponding transformed key point is minimized.

[0010] The processing circuit is further configured to: determine the relative pose of the mobile agent based on multiple feature vectors and positions of the multiple transformed keypoints and multiple feature vectors and positions of the multiple target points, for the multiple relative pose candidates. Since keypoints are a subset of all points in the source point cloud, a data processing apparatus is provided for determining relative poses in an efficient and accurate manner.

[0011] In another possible implementation, to determine the relative pose of the mobile agent, the processing circuitry is configured to: determine a point-by-point similarity metric for each point pair based on a feature vector similarity metric and a positional similarity metric. Furthermore, the processing circuitry is configured to: determine a likelihood metric for each of the plurality of relative pose candidates based on the plurality of point-by-point similarity metrics of the plurality of point pairs. The processing circuitry is also configured to: determine the relative pose of the mobile agent based on the plurality of likelihood metrics of the plurality of relative pose candidates. This enables the determination of the relative pose in an efficient and accurate manner.

[0012] In another possible implementation, the processing circuitry is used to determine the relative pose as either the relative pose candidate with the highest likelihood metric or a weighted average of two or more relative pose candidates with the highest likelihood metric, based on the plurality of likelihood metrics of the plurality of relative pose candidates. The weights of the weighted average of the respective relative pose candidates can be based on the respective likelihood metric of the respective relative pose candidate. This enables the most probable relative pose to be determined in an efficient and accurate manner.

[0013] In another possible implementation, the processing circuitry is configured to determine the likelihood metric for each of the plurality of relative pose candidates as a weighted average of the plurality of point-by-point similarity metrics for the plurality of point pairs. This allows for a more accurate determination of the likelihood metric for each of the plurality of relative pose candidates because some point pairs may be more relevant than others for evaluating the corresponding relative pose candidates.

[0014] In another possible implementation, one or more weights of the weighted average of the multiple point-by-point similarity measures of the multiple point pairs are defined by the scores of keypoints in the point pairs. Since the scores of keypoints can be determined by a neural network, higher scores may be statistically more relevant and can more accurately determine the relative pose.

[0015] In another possible implementation, the processing circuitry is configured to determine the point-to-point similarity metric for each point pair as the product of the feature vector similarity metric and the positional similarity metric. Determining the point-to-point similarity metric as the product of the feature vector similarity metric and the positional similarity metric provides a more accurate estimate of the point-to-point similarity metric, and therefore also a more accurate estimate of the relative pose. This is because the feature vector similarity metric and the positional similarity metric are complementary; for example, a “bad” pose candidate may still have a high likelihood in the sense of positional similarity, but in this case, the feature vector similarity metric may be small. For a correct pose assumption, both likelihoods are high.

[0016] In another possible implementation, the processing circuitry is configured to determine the feature vector similarity metric for each pair of points as the cosine similarity between the feature vector of the keypoint in the pair and the feature vector of the target point. This enables computationally efficient determination of the feature vector similarity metric.

[0017] In another possible implementation, the processing circuitry is used to determine the positional similarity metric for each point pair based on the Euclidean distance between the keypoint and the target point in the point pair. This enables computationally efficient determination of the positional similarity metric.

[0018] In another possible implementation, the processing circuitry is configured to: determine the positional similarity metric for each pair of points as the value of the Euclidean likelihood function of the Euclidean distance between the keypoint and the target point in the pair. Using the Euclidean likelihood function based on Euclidean distance allows for a more accurate determination of the positional similarity metric.

[0019] In another possible implementation, the Euclidean likelihood function is parameterized by one or more adjustable parameters, and the processing circuitry is used to determine these adjustable parameters. These adjustable parameters can be learned through a training process that adapts to the statistics of the source point cloud. Thus, a more accurate Euclidean likelihood function is provided.

[0020] In another possible implementation, the processing circuitry is used to implement a neural network, wherein the neural network is used to determine the feature vector and the score for each point in the source point cloud. In another possible implementation, the neural network is also used to determine the feature vector for each point in the target point cloud. This enables the efficient and accurate determination of the feature vector and score. In yet another possible implementation, the processing circuitry is used to train the neural network based on contrastive learning.

[0021] In another possible implementation, the processing circuitry selects the plurality of keypoints as a subset of the plurality of points in the source point cloud that have the highest scores. In yet another possible implementation, the processing circuitry filters the plurality of points in the source point cloud using a non-maximum suppression (NMS) scheme. This enables the selection of keypoints from different spatial regions of the source point cloud, thereby determining the relative pose more accurately.

[0022] In another possible implementation, the processing circuitry is further configured to align the source point cloud with the target point cloud based on the determined relative pose. This enables the fusion of the source and target point clouds, and the fused data can be used for further applications.

[0023] According to the second aspect, a data processing method is provided for determining a relative pose between a first pose associated with a source point cloud and an unknown second pose associated with a target point cloud.

[0024] The data processing method includes determining the feature vector and score of each point in the source point cloud, and the feature vector of each point in the target point cloud.

[0025] Furthermore, the data processing method includes: selecting multiple key points in the source point cloud based on the score of each point in the source point cloud.

[0026] The data processing method further includes: for each of the multiple relative pose candidates, transforming the multiple key points in the source point cloud from the first pose to the unknown second pose based on the corresponding relative pose candidate, so as to obtain multiple transformed key points.

[0027] Furthermore, the method includes: pairing each transformed keypoint with a target point in the target point cloud to obtain multiple point pairs, wherein the Euclidean distance from the corresponding target point to the corresponding transformed keypoint is minimized.

[0028] The method further includes: determining the relative pose of the mobile agent based on multiple feature vectors and positions of the multiple transformed key points and multiple feature vectors and positions of the multiple target points for the multiple relative pose candidates.

[0029] In another possible implementation, determining the relative pose of the mobile agent includes: determining a point-by-point similarity metric for each point pair based on feature vector similarity metrics and positional similarity metrics; determining a likelihood metric for each of the plurality of relative pose candidates based on the plurality of point-by-point similarity metrics of the plurality of point pairs; and determining the relative pose based on the plurality of likelihood metrics of the plurality of relative pose candidates.

[0030] The method provided in the second aspect of the present invention can be executed by the device provided in the first aspect of the present invention. Furthermore, other features of the method provided in the second aspect are directly derived from the functionality of the data processing apparatus provided in the first aspect and its various implementations described above and below. The other features and implementations of the method provided in the second aspect correspond to the features and implementations of the data processing apparatus provided in the first aspect.

[0031] According to a third aspect, a computer program product is provided, the computer program product including a computer-readable storage medium for storing program code, which, when executed by a computer or processor, causes the computer or processor to perform the method according to the second aspect.

[0032] The following drawings and description illustrate one or more embodiments in detail. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0033] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which:

[0034] Figure 1 A schematic diagram of the data processing apparatus provided in the embodiment is shown;

[0035] Figure 2 A schematic diagram of multiple processing blocks implemented by a data processing device according to an embodiment is shown;

[0036] Figure 3 A schematic diagram of a plurality of processing blocks including a deep neural network, implemented by a data processing device, is shown in the embodiment.

[0037] Figure 4 An exemplary Euclidean likelihood function implemented by a data processing apparatus is shown in the embodiment;

[0038] Figure 5 A flowchart of a data processing method for determining relative pose provided in an embodiment is shown;

[0039] Figure 6 Charts illustrating some exemplary results of the performance of the data processing apparatus and method provided in the embodiments are shown.

[0040] In the following text, the same reference numerals refer to the same or at least functionally equivalent features. Detailed Implementation

[0041] In the following description, reference is made to the accompanying drawings, which form part of this invention, which illustrate specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed as limiting, and the scope of the invention is defined by the appended claims.

[0042] For example, it should be understood that the disclosure relating to the described method also applies to the corresponding device or system used to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units each perform one or more of a plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a particular apparatus is described based on one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., a step to perform the function of the one or more units, or multiple steps each performing the function of one or more of the plurality of units), even if such one or more steps are not explicitly described or illustrated in the figures. Furthermore, it should be understood that, unless otherwise expressly stated, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0043] Figure 1 A schematic diagram of a data processing apparatus 100 provided in an embodiment is shown. As will be described in more detail below, the data processing apparatus 100 is used to determine a relative pose 160 of a mobile agent (e.g., a vehicle) between a first pose associated with a source point cloud 140 and an unknown second pose associated with a target point cloud 150. In one embodiment, the data processing apparatus 100 may be implemented, for example, in an autonomous vehicle or an ADAS system for a vehicle. In one embodiment, the source point cloud 140 may have already been obtained by one or more lidar sensors of the vehicle in the first pose. In one embodiment, the target point cloud 150 may be, for example, map data, such as GPS data obtained using the unknown second pose.

[0044] Before describing several embodiments of the data processing apparatus 100 in more detail, the following definitions will be given for the terms used in this invention.

[0045] Point cloud: A 3D set of points in Euclidean space.

[0046] Pose: A 6D vector consisting of 3D position coordinates (x, y, z) and three directional angles. Pose can be converted into a rotation matrix R (3×3) and a translation vector t (3×1).

[0047] Source point cloud: The point cloud captured using the first pose A.

[0048] Target point cloud: A point cloud captured using the second pose B, where the first pose A and the second pose B are close to each other, such that both point clouds are based on observations of the same environment.

[0049] Relative pose: A rigid transformation from one coordinate system / pose to another, so as to align the transformed source point cloud with the target point cloud after the source point cloud has been transformed.

[0050] Observational likelihood function: A function that evaluates the likelihood (quality measure) of a pose hypothesis. A higher likelihood value means that the pose hypothesis is more likely to be correct.

[0051] Training process: In deep learning models (e.g., neural network architectures), parameters are estimated / trained using a dataset and a loss function.

[0052] Online deployment: Using a deep learning model with fixed parameters that has been learned, i.e., applying it to an online application.

[0053] Loss function: A function that evaluates how well the model estimate fits the target value.

[0054] Loss: A scalar value representing the result of the loss function.

[0055] like Figure 1 As shown, the data processing device 100 includes processing circuitry 110 for processing data. Processing circuitry 110 can be implemented in hardware and / or software. The hardware can include digital circuitry, or both analog and digital circuitry. Digital circuitry can include components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or one or more general-purpose processors. Figure 1 As shown, the data processing device 100 may further include a communication interface 120 for exchanging data (e.g., with a lidar sensor) and a memory 130. In one embodiment, the memory 130 may be used to store executable program code that, when executed by the processing circuitry 110, causes the data processing device 100 to perform the functions and operations described herein.

[0056] Figure 2Several processing blocks, which may be implemented by the processing circuitry 110 of the data processing device 100 in the embodiment, are shown for determining the relative pose 160 of a mobile agent (e.g., a vehicle) between a first pose associated with a source point cloud 140 and an unknown second pose associated with a target point cloud 150.

[0057] The processing circuit 110 of the data processing device 100 is used to determine the feature vector and score of each point in the source point cloud 140 associated with the first pose, and the feature vector of each point in the target point cloud 150 associated with the unknown second pose, such as Figure 2 Processing blocks 111 and 112 are shown in the diagram. Further reference... Figure 3 In one embodiment, the processing circuitry 110 of the data processing apparatus 100 may implement or operate one or more deep neural networks 111a, 112a to determine multiple feature vectors 311a and scores 311b for each point in the source point cloud 140, and multiple feature vectors for multiple points in the target point cloud 150. Figure 3 The upper half of the diagram shows the processing of the source point cloud 140, while the lower half shows the processing of the target point cloud 150. In one embodiment, the processing circuitry 110 of the data processing apparatus 100 is used to train one or more neural networks 111a, 112a based on contrastive learning. As will be understood, contrastive learning is a machine learning technique that learns general features of a dataset without labels by teaching the model which data points are similar or different.

[0058] To train one or more neural networks 111a, 112a, backpropagation and gradient descent techniques can be used to optimize (a) the neural network parameters and (b) the adjustable parameters of the Euclidean likelihood function (which will be explained below). Figure 4 (described in more detail in the context of the above). As mentioned above, contrastive learning can be used to train the model, where the likelihood of the correct pose should be close to 1 and the likelihood of the incorrect pose hypothesis should be close to 0. To train the model, the processing circuit 110 may perform the following steps: (i) obtain the likelihood value l_true of the ground truth relative to the pose; (ii) sample a set of negative pose hypotheses and compute their likelihood l_false^((i)), i = 1, 2, ..., M; (iii) define the training loss function using the following formula:

[0059]

[0060] from Figure 3 It can be seen that the source point cloud 140 and the target point cloud 150 can be regarded as corresponding sets or matrices of N×3 values, that is, as the position coordinates of points in the source point cloud 140 and the target point cloud 150, respectively. Although in Figure 3In the illustrated embodiment, the source point cloud 140 and the target point cloud 150 contain the same number of points, i.e., N. However, in a further embodiment, the number of points in the source point cloud 140 may differ from the number of points in the target point cloud 150. The matrix defined by the coordinates of the points in the source point cloud 140 is the input to one or more neural networks 111a, while the matrix defined by the coordinates of the points in the source point cloud 140 is the input to one or more neural networks 112a. As described above, the output of one or more neural networks 111a processing the coordinates of the points in the source point cloud 140 is a plurality of feature vectors 311a and a plurality of scores 311b, while the output of one or more neural networks 112a processing the coordinates of the points in the target point cloud 150 is a plurality of feature vectors 312a. As used herein, the feature vector of a point is an n-dimensional vector (each dimension / component is a scalar value), where each dimension describes some feature of the point and the distribution of points around it (e.g., it may represent sharpness, the direction of surface normals, curvature). The meaning of each dimension of the learned feature is not easily interpreted by humans because it is learned through a deep learning scheme implemented by neural networks 111a and 112a, rather than being manually designed. The score is a scalar value that describes the importance of the likelihood function (its positive contribution according to the loss function).

[0061] exist Figure 3 In the illustrated embodiment, each feature vector 311a, 312a has 64 components. In another embodiment, each feature vector 311a, 312a may include more or fewer than 64 components, such as 16, 32, or 128 components. In one embodiment, the components of feature vectors 311a, 312a may have values ​​in the range [-1, 1]. In one embodiment, multiple fractions 311b of points in the source point cloud 140 may be positive values ​​in the range of 0 to 1. Although in Figure 3 The designations are neural networks 111a and 112a, but in one embodiment, a single neural network may be implemented by processing circuitry 110 to determine feature vectors 311a and 311b of points in source point 140 and feature vectors 312a of points in target point cloud 150.

[0062] exist Figure 2 and Figure 3In another processing stage 113, the processing circuit 110 of the data processing device 100 selects multiple keypoints in the source point cloud 140 based on the score 311b of each point in the source point cloud 140. In one embodiment, only these keypoints in the source point cloud 140 (along with their coordinates 313, feature vectors 315a, and scores 315b) will be used by the processing circuit 110 in subsequent processing steps. In one embodiment, the number of keypoints selected may be substantially less than the number of points in the source point cloud 140. In one embodiment, the processing circuit 110 of the data processing device 100 selects multiple keypoints as a subset of multiple points in the source point cloud 140 that have the highest scores. For example, in the case where the source point cloud 140 has 100,000 points, the processing circuit 110 of the data processing device 100 may select the 100 points in the source point cloud 140 that have the highest scores as keypoints. In one embodiment, the processing circuitry 110 of the data processing apparatus 100 is used to filter multiple points with the highest scores in the source point cloud 110 using a non-maximum suppression (NMS) scheme to avoid all key points being concentrated in the same region. In one embodiment, if a point is not a local maximum within a sphere of radius r, NMS changes its score to zero.

[0063] Therefore, after selection phase 113, the following data can be used for processing circuit 110: (a) key points in source point cloud 140 and their coordinates x_1,…〖,x〗_K 313, point-by-point feature vectors f_1,…,f_K 315a and fractional values ​​s_1,…,s_K315b, and (b) points in target point cloud 150 and their coordinates y_1,…〖,y〗_N 314 and point-by-point feature vectors g_1,…,g_N312a.

[0064] exist Figure 2 In another processing stage 114 shown, the processing circuit 110 of the data processing device 100 is used for each of the plurality of relative pose candidates (also known as pose assumptions) (e.g. Figure 2The exemplary relative pose candidate 145 shown transforms the coordinates 313 of multiple keypoints in the source point cloud 140 from a first pose associated with the source point cloud 140 to an unknown second pose associated with the target point cloud 150, based on the corresponding relative pose candidate, to obtain multiple transformed keypoints for each of the multiple relative pose candidates. In one embodiment, transforming multiple keypoints in the source point cloud 140 from a first pose associated with the source point cloud 140 to an unknown second pose associated with the target point cloud 150 based on the corresponding relative pose candidate may include rotation and / or translation. For example, in one embodiment, all keypoints can be transformed using relative pose candidate 145 by rotation R and translation t: 〖x'〗_i=Rx_i+t.

[0065] exist Figure 2 In another processing stage 115, the processing circuit 110 of the data processing device 100 is used to pair each transformed keypoint in the source point cloud 140 with a corresponding target point in the target point cloud 150 to obtain multiple point pairs. Therefore, each pair includes a transformed keypoint in the source point cloud 140 and a point in the target point cloud 150, wherein the Euclidean distance from the corresponding position of the corresponding target point to the corresponding position of the corresponding transformed keypoint is minimized. In other words, in one embodiment, the processing circuit 110 is used to find the index m(i) of the nearest neighbor point in the target point cloud 150 for each keypoint 〖x'〗_i in the source point cloud 140, such that m(i) = 〖argmin〗_j||〖x'〗_i–y_j||.

[0066] In the final processing stage, the processing circuit 110 of the data processing device 100 is used to determine the relative pose 160 of the mobile agent based on multiple feature vectors 311a and positions (i.e., coordinates 313 of multiple transformed keypoints in the source point cloud 140) and multiple feature vectors 312a and positions (i.e., coordinates 314 of multiple target points in the multiple target point clouds 150) for multiple relative pose candidates (e.g., exemplary relative pose candidate 145). Figure 2 In the illustrated embodiment, the processing circuit 110 of the data processing device 100 is used to implement the final processing stage through processing blocks 116, 117 and 118.

[0067] exist Figure 2 In processing block 116, processing circuit 110 of data processing device 100 is used to determine a point-by-point similarity metric for each point pair based on feature vector similarity metric and positional similarity metric. For example... Figure 2 As shown, in one embodiment, the processing circuit 110 of the data processing device 100 can determine the point similarity measure as the product of the feature vector similarity measure and the positional similarity measure.

[0068] In one embodiment, the processing circuit 110 of the data processing apparatus 100 is used to determine the corresponding feature vector similarity metric for each point pair as the normalized cosine similarity between the feature vectors of the corresponding keypoints in the corresponding point pair and the feature vectors of the corresponding target points, which depends on the "angle" between the feature vectors of the corresponding keypoints in the source point cloud 140 and the feature vectors of the paired target points in the target point cloud 150, the angle value of which can be in the range of 0 to 1. For example, in one embodiment, the processing circuit 110 is used to determine the corresponding feature vector similarity metric based on the following equation:

[0069]

[0070] In one embodiment, a normalization in the range [0,1] can be used as l_i^feat = 0.5(sim+1).

[0071] In one embodiment, the processing circuit 110 of the data processing device 100 is used to determine a corresponding positional similarity metric for each point pair based on the Euclidean distance between the position 313 of the corresponding key point and the position 314 of the corresponding target point in the corresponding point pair. In one embodiment, the processing circuit is used to determine the corresponding positional similarity metric for each point pair as the value of the Euclidean likelihood function of the Euclidean distance between the position 313 of the corresponding key point and the position 314 of the corresponding target point in the corresponding point pair. In one embodiment, the Euclidean likelihood function is parameterized by one or more adjustable parameters, and the processing circuit 110 of the data processing device 100 is used to determine and adjust one or more adjustable parameters.

[0072] Therefore, in one embodiment, the processing circuit 110 is used to determine the positional similarity metric (also known as the Euclidean likelihood value) l_i^euc based on the following equation:

[0073]

[0074] d = |(|x_i–y_m(i)|)|

[0075] Where d represents the Euclidean distance, and v and σ represent the learnable parameters of the Euclidean likelihood function l_i^euc, which can be learned through the training process. Figure 4 An exemplary learned Euclidean likelihood function implemented by a data processing device 100, as provided in an embodiment, is illustrated. From Figure 4 It can be seen that for d=0, the likelihood is 1, but as the distance between point pairs increases, the likelihood gradually decreases.

[0076] exist Figure 2In processing block 117, processing circuit 110 of data processing device 100 is used to determine a corresponding overall likelihood metric for each of the plurality of relative pose candidates 145 based on a plurality of point-by-point similarity metrics of a plurality of point pairs. In one embodiment, processing circuit 110 of data processing device 100 is used to determine the corresponding overall likelihood metric of each of the plurality of relative pose candidates 145 as a weighted average of a plurality of point-by-point similarity metrics of a plurality of point pairs. Processing circuit 110 may use the corresponding score of the corresponding keypoint in the corresponding point pair as the corresponding weight of the weighted average of the plurality of point-by-point similarity metrics of the plurality of point pairs.

[0077] exist Figure 2 In processing block 118, processing circuitry 110 of data processing device 100 is used to determine the relative pose 160 of a mobile agent based on multiple overall likelihood metrics of multiple relative pose candidates 145. In one embodiment, processing circuitry 110 of data processing device 100 may be used to determine the relative pose 160 as the relative pose candidate 145 with the maximum likelihood metric value based on multiple overall likelihood metrics of multiple relative pose candidates. In another embodiment, processing circuitry 110 of data processing device 100 may be used to determine the relative pose 160 as a weighted average of two or more relative pose candidates 145 with the maximum likelihood metric value. In one embodiment, the weight of the respective relative pose candidate in the weighted average is based on the respective overall likelihood metric of the respective relative pose candidate. As will be understood, to ensure robust pose estimation, an overall likelihood metric may be computed for many sampled relative pose candidates. In one example, processing circuitry 110 can be used to determine relative pose 160 by determining the overall likelihood of all relative pose candidates 145 based on the following equation:

[0078]

[0079] That is, the weighted average of the pointwise likelihood and the corresponding score s_i is used as the weighting factor, and the candidate relative pose 145 with the largest overall score is selected as the relative pose 160.

[0080] Based on the determined relative pose 160, the processing circuit 110 of the data processing device 100 can align the source point cloud 140 with the target point cloud 150, that is, transform the points of the source point cloud 140 from the first pose to the second pose.

[0081] Figure 5This is a flowchart of the steps of a method 500 for determining a relative pose 160 of a mobile agent (e.g., a vehicle) between a first pose associated with a source point cloud 140 and an unknown second pose associated with a target point cloud 150. The data processing method 500 includes step 501, which determines a feature vector 311a and a score 311b for each point in the source point cloud 140, and a feature vector 312a for each point in the target point cloud 150. In a subsequent step 503, multiple keypoints in the source point cloud 140 are selected based on the scores of each point in the source point cloud 140. The method 500 includes a further step 505, which, for each of a plurality of relative pose candidates 145, transforms the multiple keypoints in the source point cloud 140 from the first pose to an unknown second pose based on the corresponding relative pose candidate 145, to obtain multiple transformed keypoints. In another step 507, each transformed keypoint is paired with a target point in the target point cloud 150 to obtain multiple point pairs, wherein the distance from the corresponding target point to the corresponding transformed keypoint is minimized (i.e., the corresponding target point is the nearest neighbor of the corresponding transformed keypoint). Furthermore, method 500 includes step 509, which determines the relative pose 160 of the mobile agent for multiple relative pose candidates 145 based on multiple feature vectors 315a and positions of multiple transformed keypoints and multiple feature vectors 312a and positions 314 of multiple target points.

[0082] As described above in the context of the data processing apparatus 100, step 509 of determining the relative pose 160 of the mobile agent may include the following steps: (a) determining a point-by-point similarity measure for each point pair based on a feature vector similarity measure and a positional similarity measure; (b) determining a likelihood measure for each of the plurality of relative pose candidates 145 based on a plurality of point-by-point similarity measures of the plurality of point pairs; and (c) determining the relative pose 160 based on a plurality of likelihood measures of the plurality of relative pose candidates 145.

[0083] To limit computational complexity, relative pose candidates can be restricted to a defined range, such as within [–2, 2] meters. In one embodiment, the data processing apparatus can determine a relative pose 160 for correcting GPS or time tracking errors. For a relatively small range of relative pose candidates, sampling-based schemes can be used to perform robust localization, such as particle filtering, particle swarm optimization, or differential evolution. For example, differential evolution (DE) is a method for optimizing a problem by iteratively attempting to improve candidate schemes (relative poses in the embodiments disclosed herein) with respect to a given quality metric (likelihood).

[0084] Figure 6Graphs illustrating some exemplary results of the performance of the data processing apparatus 100 and method 500 provided in the embodiments for estimating the relative pose between lidar scans are shown. In this example, a trained likelihood function is used in a differential evolution method to search for the correct relative pose.

[0085] Figure 6 The results shown are based on a dataset containing 10 driving sequences (for each frame of each sequence), LiDAR scans (point clouds), and corresponding global pose ground truth values. Figure 6 The results shown indicate that multiple sequences from the dataset have been used to train neural networks 111a and 112a, and another sequence has been used to test performance. Source-target point cloud pairs have been extracted from the odometry dataset, where source point cloud 140 is a LiDAR scan of each frame, and target point cloud 150 is a LiDAR scan of another frame, with their relative poses within a 5-meter radius. First, the source coordinates are transformed to target coordinates based on a ground truth table, and then transformed again using another random relative pose. The aim of this experiment is to estimate the random relative pose using a sampling-based method that employs a likelihood function learned by the data processing apparatus 100 provided in the embodiment. For example, differential evolution (DE) methods have been used to search for the optimal relative pose with the highest likelihood value. Population sizes of 50 and 64 generations were used.

[0086] Random relative poses have been sampled within a uniform sampling range of roll / pitch / yaw (direction) and x / y / z (position), with lower bounds of [–1,–1,–5] degrees and [–5,–5,–1] meters, and upper bounds of [1,1,5] degrees and [5,5,1] meters. For comparison, two settings of the likelihood function have been used: (a) a first setting where 64 randomly selected points in the source point cloud 140 are used with the sum of point-wise Euclidean likelihoods, and (b) a second setting where 64 learned keypoints in the source point cloud 140 are used with a weighted average of point-wise likelihoods, where the point-wise likelihood is the product of Euclidean likelihood (i.e., a positional similarity measure) and eigenvalue likelihood (i.e., an eigenvector similarity measure). While the first setting corresponds to a conventional method, the second setting is an embodiment of the data processing apparatus 100.

[0087] for Figure 6 The results shown in the table have been compared using the rotation and translation errors for estimating the relative pose. For rotation and translation errors, Figure 6 The table lists the percentages below the threshold (i.e., what percentage of the samples are below a certain threshold; higher means better) as well as the mean, median, and maximum error (lower means better). These results demonstrate that the embodiment of the data processing device 100 exhibits improved performance.

[0088] Those skilled in the art will understand that the “blocks” (“units”) of various diagrams (methods and apparatuses) represent or describe the functionality of an embodiment (and are not necessarily individual “units” in hardware or software), thereby equivalently describing the functionality or features of an apparatus embodiment and a method embodiment (unit = step).

[0089] Regarding the embodiments disclosed herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the embodiments of the described apparatus are merely exemplary. For example, the unit division is merely a logical functional division, and in actual implementation, it can be another division. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or described can be implemented through some interface. Direct coupling or communication connection between devices or units can be implemented electronically, mechanically, or otherwise.

[0090] The units described as discrete components may or may not be physically separate. The components shown as units may or may not be physical units, and may be located in one location or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0091] Furthermore, the functional units in the embodiments disclosed herein can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

Claims

1. A data processing apparatus (100), characterized in that, For determining the relative pose (160) of a mobile agent between a first pose associated with a source point cloud (140) and an unknown second pose associated with a target point cloud (150), wherein the data processing device (100) includes a processing circuit (110) for: The feature vector (311a) and score (311b) of each point in the source point cloud (140) and the feature vector (312a) of each point in the target point cloud (150) are determined (111, 112); wherein the feature vector (311a) and score (311b) of each point in the source point cloud (140) and the feature vector (312a) of each point in the target point cloud (150) are determined by implementing neural networks (111a, 112a). Based on the score (311b) of each point in the source point cloud (140), select (113) a plurality of key points in the source point cloud (140); wherein the plurality of key points are selected as a subset of the plurality of points in the source point cloud (140) with the highest scores; For each of the multiple relative pose candidates (145), based on the corresponding relative pose candidate (145), the multiple key points in the source point cloud (140) are transformed from the first pose transformation (114) to the unknown second pose to obtain multiple transformed key points; the relative pose candidates are generated based on sampling schemes, including particle filtering, particle swarm optimization or differential evolution. Each transformed key point is paired with a target point in the target point cloud (150) (115) to obtain multiple point pairs, wherein the distance from the corresponding target point to the corresponding transformed key point is the smallest; For the multiple relative pose candidates (145), based on the multiple feature vectors (315a) and positions of the multiple transformed key points and the multiple feature vectors (312a) and positions (314) of the multiple target points, the relative pose of the mobile agent is determined (118) (160). In order to determine the relative pose (160) of the mobile agent, the processing circuit (110) is used to: Based on feature vector similarity and position similarity, a point-by-point similarity metric is determined for each point pair; based on the multiple point-by-point similarity metrics of the multiple point pairs, a likelihood metric is determined for each of the multiple relative pose candidates (145); based on the multiple likelihood metrics of the multiple relative pose candidates, the relative pose (160) of the mobile agent is determined; wherein, the point-by-point similarity metric of each point pair is determined as the product of the feature vector similarity metric and the position similarity metric, the feature vector similarity metric of each point pair is determined as the cosine similarity between the feature vector (315a) of the key point in the point pair and the feature vector (312a) of the target point, and the position similarity metric of each point pair is determined based on the Euclidean distance between the key point and the target point in the point pair; The processing circuit (110) is used to determine the positional similarity metric for each point pair based on the Euclidean distance between the key point and the target point in the point pair; The processing circuit (110) is used to determine the positional similarity metric of each point pair as the value of the Euclidean likelihood function of the Euclidean distance between the key point and the target point in the point pair.

2. The data processing apparatus (100) according to claim 1, characterized in that, The processing circuit (110) is used to determine the relative pose (160) as a weighted average of two or more relative pose candidates (145) with the largest likelihood measure, based on the plurality of likelihood measures of the plurality of relative pose candidates (145), wherein the weight of the corresponding relative pose candidate in the weighted average is based on the plurality of likelihood measures of the corresponding relative pose candidate (145).

3. The data processing apparatus (100) according to claim 1, characterized in that, The processing circuit (110) is used to determine the likelihood metric of each of the plurality of relative pose candidates (145) as a weighted average of the plurality of point-by-point similarity metrics of the plurality of point pairs.

4. The data processing apparatus (100) according to claim 3, characterized in that, One or more weights of the weighted average of the multiple point-to-point similarity measures of the multiple point pairs are defined by the scores (315b) of the key points in the point pairs.

5. The data processing apparatus (100) according to claim 1, characterized in that, The processing circuit (110) is used to determine the positional similarity metric of each point pair as the value of the Euclidean likelihood function of the Euclidean distance between the key point and the target point in the point pair.

6. The data processing apparatus (100) according to claim 5, characterized in that, The Euclidean likelihood function is parameterized by one or more adjustable parameters, and the processing circuit (110) is used to determine the one or more adjustable parameters.

7. The data processing apparatus (100) according to claim 1, characterized in that, The processing circuit (110) is used to filter multiple points with the highest scores in the source point cloud (140) using a non-maximum suppression (NMS) scheme.

8. The data processing apparatus (100) according to any one of the preceding claims, characterized in that, The processing circuit (110) is also used to align the source point cloud (140) with the target point cloud (150) based on the determined relative pose (160) of the mobile agent.

9. A data processing method (500), characterized in that, For determining the relative pose (160) of a mobile agent between a first pose associated with a source point cloud (140) and an unknown second pose associated with a target point cloud (150), wherein the data processing method (500) includes: (501) Determine the feature vector (311a) and score (311b) of each point in the source point cloud (140), and the feature vector (312a) of each point in the target point cloud (150); wherein the feature vector (311a) and score (311b) of each point in the source point cloud (140) and the feature vector (312a) of each point in the target point cloud (150) are determined by implementing neural networks (111a, 112a). Based on the score of each point in the source point cloud (140), select (503) multiple key points in the source point cloud (140); wherein, select the multiple key points as a subset of multiple points in the source point cloud (140) with the highest score; For each of the multiple relative pose candidates (145), based on the corresponding relative pose candidate (145), the multiple key points in the source point cloud (140) are transformed from the first pose transformation (505) to the unknown second pose to obtain multiple transformed key points; the relative pose candidates are generated based on sampling schemes, including particle filtering, particle swarm optimization or differential evolution. Each transformed key point is paired with a target point in the target point cloud (150) (507) to obtain multiple point pairs, wherein the distance from the corresponding target point to the corresponding transformed key point is the smallest; For the multiple relative pose candidates (145), based on the multiple feature vectors (315a) and positions of the multiple transformed key points and the multiple feature vectors (312a) and positions (314) of the multiple target points, the relative pose of the mobile agent is determined (509) (160). Determining (509) the relative pose (160) of the mobile agent includes: Based on feature vector similarity and location similarity, a point-by-point similarity measure is determined for each point pair; Based on multiple point-to-point similarity measures of the multiple point pairs, a likelihood measure is determined for each of the multiple relative pose candidates (145). The relative pose (160) is determined based on multiple likelihood measures of the multiple relative pose candidates (145). Specifically, the point-to-point similarity metric for each point pair is determined as the product of the feature vector similarity metric and the positional similarity metric; the feature vector similarity metric for each point pair is determined as the cosine similarity between the feature vector (315a) of the key point in the point pair and the feature vector (312a) of the target point; the positional similarity metric for each point pair is determined based on the Euclidean distance between the key point and the target point in the point pair; and the positional similarity metric for each point pair is determined as the value of the Euclidean likelihood function of the Euclidean distance between the key point and the target point in the point pair.

10. A computer program product, characterized in that, Includes a computer-readable storage medium for storing program code, which, when executed by a computer or processor, causes the computer or processor to perform the method (500) according to claim 9.

Images