A method and apparatus for relocating by millimeter wave radar

Abstract

The application discloses a method and device for repositioning by millimeter wave radar. The millimeter wave radar point cloud is encoded into a description sub, the position interval thereof is obtained, and a repositioning model is obtained through training, so that repositioning is realized. The method disclosed by the application expands the use scene of the millimeter wave radar, and the millimeter wave radar is low in price, small in size, not easy to be affected by environmental factors, and low in cost, so that the repositioning task is realized more efficiently.

Description

Technical Field

[0001] This application relates to the field of radar detection technology, and in particular to a method and apparatus for relocation via millimeter-wave radar. Background Technology

[0002] As autonomous vehicles become a reality, their safety is closely related to positioning technology. When a vehicle's location is lost, relocation is required, that is, retrieving the current location when positioning fails.

[0003] Most existing technologies achieve relocalization by combining Simultaneous Localization and Mapping (SLAM) with lidar (LiDAR). However, lidar has drawbacks such as large size, susceptibility to environmental factors, and high cost, making mass production difficult. Therefore, how to achieve relocalization tasks at low cost and high efficiency has become an urgent problem to be solved. Summary of the Invention

[0004] To address the aforementioned problems, this application provides a method and apparatus for repositioning using millimeter-wave radar. The aim is to achieve efficient repositioning using millimeter-wave radar.

[0005] This application discloses a method for relocation using millimeter-wave radar, the method comprising:

[0006] Encode the sample point cloud of millimeter-wave radar into sample descriptors;

[0007] Obtain the positional spacing of the sample descriptors in the coding space;

[0008] The relocation model is trained based on the sample descriptors and the location spacing.

[0009] Relocation is performed using the relocation model obtained through training.

[0010] Optionally, relocalization can be performed using the trained relocalization model, including:

[0011] Encode the point cloud of the millimeter-wave radar into a descriptor to be located;

[0012] The relocation model is used to perform a nearest neighbor search on the descriptor to be located, and the resulting descriptor position is the relocation result.

[0013] Optionally, training a relocalization model based on the sample descriptors and the location spacing includes: using a convolutional neural network as the backbone network, and training a relocalization model based on the sample descriptors, the location spacing, and the NetVLAD algorithm.

[0014] Optionally, a relocalization model is trained based on the sample descriptors and the positional spacing, including: using TripletLoss as the loss function, the training model moves the positional spacing between anchor examples and positive examples closer together, and moves the positional spacing between anchor examples and negative examples further apart.

[0015] Optionally, the method further includes training a relocalization model based on the sample descriptors and the positional spacing, and using an online training set generation method to re-acquire the sample descriptors and the positional spacing based on the positions of different sample descriptors.

[0016] Based on the above-mentioned method for repositioning via millimeter-wave radar, this application also discloses an apparatus for repositioning via millimeter-wave radar, comprising: an encoding unit, a position spacing acquisition unit, a training unit, and a repositioning unit;

[0017] The encoding unit is used to encode the sample point cloud of the millimeter-wave radar into a sample descriptor;

[0018] The position spacing acquisition unit is used to acquire the position spacing of the sample descriptors in the coding space;

[0019] The training unit is used to train a relocalization model based on the sample descriptors and the positional spacing.

[0020] The relocation unit is used to perform relocation using the trained relocation model.

[0021] Optionally, the relocation unit includes:

[0022] The descriptor encoding unit is used to encode the point cloud of the millimeter-wave radar into a descriptor to be located;

[0023] The nearest neighbor search subunit is used to perform a nearest neighbor search on the descriptor to be located to obtain the relocation result.

[0024] Optionally, the training unit is used to: train a relocalization model based on the sample descriptors, the location spacing, and the NetVLAD algorithm, using a convolutional neural network as the backbone network.

[0025] Optionally, the training unit is used to: use TripletLoss as the loss function to train the positional distance between anchor examples and positive examples to be closer together, and to train the positional distance between anchor examples and negative examples to be farther apart.

[0026] Optionally, it also includes: a training set generation unit, used to re-acquire the sample descriptors and the position spacing according to the positions of different sample descriptors by using an online training set generation method.

[0027] This application discloses a method and apparatus for relocation using millimeter-wave radar. By encoding millimeter-wave radar point clouds into descriptors and obtaining their positional spacing, a relocation model is obtained through training, which is then used to achieve relocation. The method described in this application expands the application scenarios of millimeter-wave radar, which is inexpensive, small in size, and less affected by environmental factors, thus achieving relocation tasks more efficiently at a lower cost. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating a method for relocation using millimeter-wave radar disclosed in an embodiment of this application.

[0030] Figure 2 This is a flowchart illustrating a method for operating a relocation model based on millimeter-wave radar, as disclosed in an embodiment of this application.

[0031] Figure 3 This is a schematic diagram illustrating the training process of a millimeter-wave radar relocation model in a specific scenario, as disclosed in an embodiment of this application.

[0032] Figure 4 This is a schematic diagram of a device for repositioning via millimeter-wave radar disclosed in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] Example 1: This application discloses a method for relocation using millimeter-wave radar, applied in the field of radar detection technology. By encoding millimeter-wave radar point clouds into descriptors and obtaining their positional spacing, a relocation model is obtained through training, which is then used to achieve relocation.

[0035] For details, please refer to Figure 1 The relocation method using millimeter-wave radar disclosed in this embodiment includes the following steps:

[0036] Step 101: Encode the sample point cloud of the millimeter-wave radar into sample descriptors.

[0037] In the method described in this embodiment, as an optional method, the NetVLAD algorithm combined with a convolutional neural network (CNN) can be used for encoding.

[0038] The Vector of Locally Aggregated Descriptors (VLAD) algorithm is a classic feature encoding algorithm that can compress several local features into a global feature of a specific size, achieving feature dimensionality reduction through clustering. However, it cannot be directly embedded into a CNN. In the method described in this embodiment, NetVLAD is a differentiable improvement to the VLAD algorithm, allowing its parameters to be trained through backpropagation. Therefore, the NetVLAD layer can be embedded into a CNN to output VLAD descriptors.

[0039] The CNN part can use currently popular backbone networks. For example, point cloud data can be detected using network structure models such as CenterPoint and PointPillar for 3D object detection. When converting point clouds into depth maps or bird's-eye views, it can be achieved through Visual Geometry Group Network (VGG) and ResNet.

[0040] Step 102: Obtain the positional spacing of the sample descriptors in the coding space.

[0041] In the method described in this embodiment, as a feasible approach, the distance between the true values ​​of the positions of descriptors can be characterized by the magnitude of the Euclidean distance between them in the coding space. Euclidean distance (also known as the Euclidean metric) is a commonly used distance definition in information coding, referring to the true distance between two points in a one-dimensional space, or the natural length of a vector (i.e., the distance from that point to the origin). In two-dimensional and three-dimensional spaces, the Euclidean distance is the actual distance between two points.

[0042] In this embodiment, the method introduces TripletLoss as the loss function. Its principle is to input a triplet, including an anchor example, a positive example, and a negative example. Similarity calculation between samples is achieved by optimizing the distance between the anchor example and the positive example to be less than the distance between the anchor example and the negative example. In this embodiment, Query represents the point cloud data frame at time t, Positive represents the frame closest to Query, and Negatives represent all or some data frames whose distance from Query is greater than a certain threshold. One Query frame, one Positive frame, and several Negative frames are used as a unit in the training set. Multiple such units constitute a batch of training inputs into the model, obtaining Query descriptors, Positive descriptors, and multiple Negative descriptors corresponding to the data frames Query, Positive, and Negative.

[0043] Step 103: Train the relocalization model based on the sample descriptors and the location spacing.

[0044] In the method described in this embodiment, as a feasible approach, TripletLoss is used as the loss function, and its training method is as follows:

[0045] TripletLoss is used as the training loss function. The loss is directly proportional to the Euclidean distance between the Query descriptor and the Positive descriptor, and inversely proportional to the mean Euclidean distance between the Query descriptor and multiple Negative descriptors. During training, the loss decreases as the Euclidean distance between the Query descriptor and the Positive descriptor decreases, and the Euclidean distance between the Query descriptor and multiple Negative descriptors increases. Therefore, during model operation, the input data will be matched with the point closest to it in real space with a very high probability, thus achieving the relocalization task.

[0046] As a feasible method, the method described in this embodiment employs an online training set generation approach. After each batch of training, the input data is reconstructed based on the position of the descriptors in the coding space. The input data is generated online based on the position of the millimeter-wave radar data.

[0047] Step 104: Perform relocalization using the relocalization model obtained through training.

[0048] In the method described in this embodiment, the input data is a frame of point cloud. After the relocalization model obtains the descriptor encoded from the point cloud through the network, it performs a nearest neighbor search on the descriptor. The position of the matched descriptor is the final predicted position, which is the output result.

[0049] In the method described in this embodiment, millimeter-wave radar point clouds are encoded into descriptors, their positional spacing is obtained, and then a relocalization model is trained to achieve relocalization. The generalization ability of CNNs can solve the inherent sparsity and high noise problem of millimeter-wave radar point clouds; the NetVLAD algorithm has low computational cost, lossless information, and high retrieval accuracy, and can be embedded into deep neural networks to learn and update parameters; the training uses an online dataset generation method, which can accelerate model convergence and help millimeter-wave radar efficiently achieve relocalization tasks. Furthermore, from a practical application perspective, millimeter-wave radar is inexpensive and mass-producible, reducing relocalization costs.

[0050] Example 2: This application discloses a method for operating a relocation model based on millimeter-wave radar. Please refer to [link / reference]. Figure 2 The method described in this embodiment introduces the working process of the relocation model.

[0051] Step 201: Input a point cloud frame from the millimeter-wave radar into the relocalization model.

[0052] The point cloud of the current frame detected by the millimeter-wave radar is sent as input data to the encoding layer of the relocation model.

[0053] Step 202: The encoding layer of the model encodes the point cloud into descriptors.

[0054] As a feasible method, the method described in this embodiment uses a CNN+NetVLAD network structure to encode the received point cloud to obtain a descriptor, wherein the descriptor is a vector representation.

[0055] In the method described in this embodiment, TripletLoss is introduced as the loss function. One frame of Query, one frame of Positive, and several frames of Negative are used as a unit of the training set. Multiple such units constitute a batch of training data and are input into the model to obtain the corresponding descriptors.

[0056] Step 203: The dataset layer of the model generates a dataset online based on the location of the descriptors.

[0057] In the method described in this embodiment, the CNN+NetVLAD network structure enables NetVLAD to be generalized, and parameters are updated through backpropagation to generate input data online based on the descriptor positions of point clouds in different frames.

[0058] Step 204: The prediction layer of the model receives the dataset and makes predictions.

[0059] In the method described in this embodiment, the point closest to the Query descriptor in the actual space is predicted based on the TripletLoss loss function.

[0060] Step 205: The model outputs the predicted location of the descriptor.

[0061] In the method described in this embodiment, the input data will be matched with the point that is closest to its actual location in space (i.e., the predicted location of the descriptor) with a very high probability as the output data, thereby realizing the relocation task.

[0062] In the method described in this embodiment, the application scenarios of millimeter-wave radar are expanded by using a relocation model based on millimeter-wave radar. Moreover, millimeter-wave radar has stronger performance, smaller size, and lower cost than lidar, and can efficiently complete relocation tasks at low cost.

[0063] The method disclosed in the above embodiments can be used in a vehicle environment. Its model training method is as follows: The point cloud detected by millimeter-wave radar at time t is used as the sample point cloud, and it is encoded using a CNN+NetVLAD network structure to obtain sample descriptors. Due to the introduction of the TripletLoss loss function, the sample point cloud contains one frame Query (the data frame of the point cloud at time t), one frame Positive (the data frame whose position is closest to Query), and several frames Negative (all or some data frames whose distance from Query is greater than a certain threshold). Therefore, the encoded sample descriptor contains a Query descriptor, a Positive descriptor, and multiple Negative descriptors. After training, the model can find the position of the point closest to the Query descriptor and output it, completing the relocalization task. During vehicle operation, as time changes, the millimeter-wave radar can continuously detect point clouds. At this time, the relocalization model updates the point cloud at time t1 to the sample point cloud, thereby realizing online generation of the training set.

[0064] In the method described in this application, as an optional method, Figure 3 This is a schematic diagram illustrating the training process of a millimeter-wave radar relocation model in a specific scenario as disclosed in an embodiment of this application.

[0065] As shown in the figure, the method described in this embodiment is based on a CNN network and the k-nearest neighbor algorithm (kNN). First, the input data is constructed by finding and matching the data frame closest to the actual point cloud data. During training, features are extracted online from the dataset. Based on the position of the descriptors in the encoding space, the training / dataset is reconstructed at each training stage to accelerate model convergence. During training, point cloud data, positive examples, and negative examples are used as a unit of the training set. The relocalization model is trained using a VGG16 network + NetVLAD algorithm model structure with TripletLoss as the loss function, enabling it to output the data frame closest to the actual point cloud data. Simultaneously, the CNN + NetVLAD network structure allows backpropagation to obtain gradients for parameter adjustment. The final relocalization model is tested for accuracy by comparing the predicted values ​​with positive examples from the actual point cloud data to obtain a correctly functioning relocalization model.

[0066] The method described in this embodiment is only provided as a relocalization model training method for one scenario for reference. In actual operation, no specific limitations are made on the algorithm and network.

[0067] Based on the method for repositioning using millimeter-wave radar disclosed in the above embodiments, this embodiment correspondingly discloses a device for repositioning using millimeter-wave radar. Please refer to... Figure 4 The device for repositioning via millimeter-wave radar includes: an encoding unit 401, a position spacing acquisition unit 402, a training unit 403, and a repositioning unit 404.

[0068] The encoding unit 401 is used to encode the sample point cloud of the millimeter-wave radar into a sample descriptor.

[0069] The position spacing acquisition unit 402 is used to acquire the position spacing of the sample descriptors in the coding space;

[0070] The training unit 403 is used to train a relocation model based on the sample descriptor and the positional spacing.

[0071] The relocation unit 404 is used to perform relocation using the trained relocation model.

[0072] Optionally, the relocation unit 404 includes:

[0073] The descriptor encoding unit is used to encode the point cloud of the millimeter-wave radar into a descriptor to be located;

[0074] The nearest neighbor search subunit is used to perform a nearest neighbor search on the descriptor to be located to obtain the relocation result.

[0075] Optionally, the training unit 403 is used to: train a relocalization model based on the sample descriptors, the positional spacing, and the NetVLAD algorithm, using a convolutional neural network as the backbone network.

[0076] Optionally, the training unit 403 is used to: use TripletLoss as the loss function to train and bring the positional distance between anchor examples and positive examples closer together, and to widen the positional distance between anchor examples and negative examples.

[0077] Optionally, it also includes: a training set generation unit, used to re-acquire the sample descriptors and the positional spacing based on the positions of different sample descriptors using an online training set generation method.

[0078] The embodiments in this specification are described in a progressive manner. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant details can be found in the method section.

[0079] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0080] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0081] The features described in the embodiments of this specification can be substituted for or combined with each other, so that those skilled in the art can implement or use this application.

[0082] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for repositioning via millimeter-wave radar, characterized in that, The method includes: Encode the sample point cloud of millimeter-wave radar into sample descriptors; Obtain the positional spacing of the sample descriptors in the coding space; The relocalization model is trained based on the sample descriptors and the location spacing. Relocation is performed using the relocation model obtained through training.

2. The method according to claim 1, characterized in that, Relocalization is performed using the trained relocalization model, including: Encode the point cloud of the millimeter-wave radar into a descriptor to be located; The relocation model is used to perform a nearest neighbor search on the descriptor to be located, and the resulting descriptor position is the relocation result.

3. The method according to claim 1, characterized in that, Training a relocalization model based on the sample descriptors and the positional spacing includes: using a convolutional neural network as the backbone network, and training a relocalization model based on the sample descriptors, the positional spacing, and the NetVLAD algorithm.

4. The method according to claim 1, characterized in that, The relocalization model is trained based on the sample descriptors and the position spacing, including: using TripletLoss as the loss function, the position spacing between anchor examples and positive examples is brought closer together, and the position spacing between anchor examples and negative examples is brought further apart.

5. The method according to claim 1, characterized in that, The method further includes: using an online training set generation method to re-obtain the sample descriptors and the positional spacing based on the positions of different sample descriptors.

6. A device for repositioning via millimeter-wave radar, characterized in that, include: Encoding unit, position spacing acquisition unit, training unit, and relocalization unit; The encoding unit is used to encode the sample point cloud of the millimeter-wave radar into a sample descriptor; The position spacing acquisition unit is used to acquire the position spacing of the sample descriptors in the coding space; The training unit is used to train a relocalization model based on the sample descriptors and the positional spacing. The relocation unit is used to perform relocation using the trained relocation model.

7. The apparatus according to claim 6, characterized in that, The relocation unit includes: The descriptor encoding unit is used to encode the point cloud of the millimeter-wave radar into a descriptor to be located; The nearest neighbor search subunit is used to perform a nearest neighbor search on the descriptor to be located to obtain the relocation result.

8. The apparatus according to claim 6, characterized in that, The training unit is used to: train a relocalization model based on the sample descriptors, the positional spacing, and the NetVLAD algorithm, using a convolutional neural network as the backbone network.

9. The apparatus according to claim 6, characterized in that, The training unit is used to: use TripletLoss as the loss function to train and bring the positional distance between anchor examples and positive examples closer together, and to widen the positional distance between anchor examples and negative examples.

10. The apparatus according to claim 6, characterized in that, Also includes: The training set generation unit is used to re-acquire the sample descriptors and the positional spacing based on the positions of different sample descriptors by employing an online training set generation method.

Images