A radar odometry method fusing semantic features and related devices

Abstract

The application provides a radar odometry method fusing semantic features and related equipment, comprising: collecting multiple frames of initial point clouds of a target environment for projection to obtain a distance image, inputting the distance image into a full convolution network for semantic segmentation to obtain a semantic segmentation label; taking the semantic segmentation label as a constraint condition, distinguishing features of each frame of initial point cloud according to a curvature threshold to obtain multiple edge points and multiple plane points, and selecting multiple strong edge features and multiple strong plane features therefrom; obtaining feature lines and feature surfaces according to the multiple strong edge features and the multiple strong plane features, constructing a residual block and setting a semantic confidence score for the residual block; adaptively adjusting distances of each strong edge feature to the feature line and distances of each strong plane feature to the feature surface according to the semantic confidence score, calculating a credibility of a residual, and performing feature matching to obtain a pose according to the credibility of the residual.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a radar odometry method and related equipment that integrates semantic features. Background Technology

[0002] LiDAR sensors are widely used in robotics, drones, autonomous driving, 3D reconstruction, and many other fields, and are important sensors in many remote sensing applications. As the front end of SLAM (Simultaneous Localization and Mapping), the accuracy of LiDAR ranging (LO) is crucial.

[0003] The goal of radar odometry is to acquire environmental information using lidar and then autonomously locate itself based on that information. 3D lidar has the ability to directly acquire spatial distance information and operate at night, making it suitable for complex environments and unaffected by darkness. It has immense application value and potential in Earth observation missions, therefore improving the accuracy of odometry and minimizing its cumulative drift error is crucial.

[0004] Traditional LO methods typically use geometric features to perform point cloud registration and solve for pose information. Generally, planar points and edge points are extracted from radar point clouds as geometric features. In special environments, geometric features such as lines and cylinders can also be extracted. Compared to camera-generated LiDAR point clouds, which are sparse, unstructured, and lack texture features, they can accurately represent 3D spatial structures. With the development of deep learning, semantic segmentation of both images and LiDAR point clouds has improved.

[0005] The earliest approach to directly address the unstructured nature of point clouds through semantic segmentation was pioneered by Charles R. Qi et al., who utilized symmetric functions and developed the PointNet network model. Specifically, PointNet independently learns point-by-point features through multiple MLP layers and extracts global features using max-pooling layers. Subsequently, excellent semantic models such as DeepNet, PointNet++, and RangeNet++ were introduced. RangeNet++ employs a spherical projection method for point clouds, converting 3D point cloud data into data compatible with many mature 2D image processing methods. 2D fully convolutional semantic segmentation is then performed on this 2D image. The successfully segmented 2D information is then transferred to a 3D point cloud for further optimization of the segmentation structure. On the KITTI dataset, RangeNet++ demonstrates superior semantic segmentation compared to previous techniques.

[0006] The most classic point cloud registration algorithm in lidar ranging is the ICP (Iterative Closest Point) algorithm. The ICP algorithm uses the nearest point as the correspondence, optimizes and iteratively transforms parameters until the required accuracy is met, at which point the iteration terminates. However, this algorithm relies on initial parameters; poor initial values ​​increase the number of iterations and easily lead to local optima. To improve these shortcomings, numerous variants of the ICP algorithm have been developed. For example, Point-to-Line ICP uses the distance metric from the source point cloud to the target point cloud as the error function, reducing the number of iterations and errors, but it is less robust to large initial displacements. Other variants use the distance metric from a point to a plane or from a plane to a plane as the error function. However, these are based on geometric features, and judging geometric features by simply sampling surrounding points to simulate curvature information easily introduces a large amount of error. Summary of the Invention

[0007] This invention provides a radar odometry method and related equipment that integrates semantic features, with the aim of improving the accuracy of lidar ranging.

[0008] To achieve the above objectives, the present invention provides a radar odometry method that integrates semantic features, comprising:

[0009] Step 1: Collect multiple frames of initial point cloud data of the target environment using a lidar sensor;

[0010] Step 2: Project the initial point cloud of each frame to obtain the distance image of the initial point cloud of each frame, and input the distance image into the fully convolutional network for semantic segmentation to obtain the semantic segmentation label of the initial point cloud of each frame.

[0011] Step 3: Using semantic segmentation labels as constraints, perform feature differentiation on the initial point cloud of each frame according to the curvature threshold to obtain multiple edge points and multiple planar points, and select multiple strong edge features from the multiple edge points and multiple strong planar features from the multiple planar points.

[0012] Step 4: For each edge point in the initial point cloud of each frame, traverse each strong edge feature in the edge points in turn, search for edge points in the edge points of the previous frame initial point cloud that are less than a preset distance from each strong edge feature in the current frame initial point cloud and fit them into feature lines.

[0013] Step 5: For each plane point in the initial point cloud of each frame, traverse each strong plane feature in the plane point in turn, search for plane points in the plane points of the previous frame initial point cloud whose distance to each strong plane feature in the current frame initial point cloud is less than a preset distance, and fit them into a feature surface.

[0014] Step 6: Construct residual blocks using the distance constraints of feature lines and feature surfaces, and set semantic confidence scores for the residual blocks using the semantic labels of feature lines or feature surfaces;

[0015] Step 7: Adaptively adjust the distance from each strong edge feature to the feature line and the distance from each strong plane feature to the feature surface based on the semantic confidence score, calculate the confidence of the residual, and perform feature matching based on the confidence of the residual to obtain the pose.

[0016] Furthermore, the distance image is obtained by projecting multiple initial point clouds, including:

[0017] The image coordinates are obtained by spherically projecting the initial point clouds of multiple frames according to the following formula:

[0018]

[0019] Among them, among them, Represents the pixel coordinates of the image. The Euclidean distance is the initial point cloud to the origin of the lidar sensor. This represents the vertical viewing angle range of the radar, indicating the vertical viewing angle range of the radar. , This indicates the downward field of view range in the vertical direction of the radar. For the height of the 2D ranging image, Represents the coordinates of a 3D point cloud. The width of the 2D ranging image;

[0020] The distance image is obtained based on the image coordinates.

[0021] Furthermore, the distance image is input into a fully convolutional network for semantic segmentation to obtain semantic segmentation labels for the initial point cloud of each frame, including:

[0022] The distance image is input into a fully convolutional network for semantic segmentation to obtain semantic labels for the pixels in the distance image. The fully convolutional network includes an hourglass encoder and a decoder.

[0023] By mapping the initial point cloud to the pixels of the distance image, semantic labels are assigned to the initial point cloud, thus obtaining the semantic segmentation label for each frame of the initial point cloud.

[0024] Furthermore, step 3 includes:

[0025] The curvature of the initial point cloud in each frame is:

[0026]

[0027] in, This represents an initial point cloud frame. , , They represent the first Initial point cloud of frame, first Initial point cloud of frame, , , These are two initial point cloud frames acquired from the same wiring harness. Indicates the first The 3D coordinates of the initial point cloud in frame c, where c represents the first frame. The curvature of the point cloud at the beginning of the frame. Indicates the first Semantic segmentation labels for the initial point cloud of the frame;

[0028] Set a curvature threshold a, and perform feature differentiation on the initial point cloud of each frame based on the curvature threshold a. Points in the initial point cloud of each frame with curvature C greater than the curvature threshold a are taken as edge points, and points in the initial point cloud of each frame with curvature C less than the curvature threshold a are taken as plane points.

[0029] The initial point cloud in each frame, which is collected by the same wire harness, is divided into 6 parts;

[0030] For each part of the initial point cloud, with semantic segmentation labels as constraints, Ne strong edge features are selected from the edge points of the initial point cloud, and Np strong planar features are selected from the planar points of each part.

[0031] Furthermore, step 4 includes:

[0032] For each edge point in the initial point cloud of each frame, each strong edge feature in the edge points is traversed sequentially. Using a K-tree, multiple edge points in the edge points of the previous frame's initial point cloud that are less than a preset distance from each strong edge feature in the current frame's initial point cloud are searched. Then, a feature line is fitted using principal component analysis.

[0033] Furthermore, step 5 includes:

[0034] For each planar point in the initial point cloud of each frame, each strong planar feature in the planar point is traversed sequentially. Using a K-tree, planar points in the planar points of the previous frame's initial point cloud that are less than a preset distance from each strong planar feature in the current frame's initial point cloud are searched. Then, a feature surface is fitted by principal component analysis.

[0035] Furthermore, the semantic confidence score is:

[0036]

[0037] in, The semantic segmentation label used to determine whether strong edge features or strong planar features are the same as the kth point among the N nearest edge points searched.

[0038] Furthermore, the process before step 2 also includes:

[0039] Point cloud filtering is performed on the initial point cloud of multiple frames to remove invalid points, resulting in the filtered point cloud of multiple frames.

[0040] The distance image is obtained by projecting the filtered point cloud from multiple frames.

[0041] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a radar odometry method that fuses semantic features.

[0042] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a radar odometry method that fuses semantic features.

[0043] The above-described solution of the present invention has the following beneficial effects:

[0044] This invention acquires multiple frames of initial point clouds of the target environment using a lidar sensor, projects them to obtain distance images for each frame of the initial point cloud, and inputs these distance images into a fully convolutional network for semantic segmentation to obtain semantic segmentation labels for each frame of the initial point cloud. Using these semantic segmentation labels as constraints, feature discrimination is performed on each frame of the initial point cloud based on a curvature threshold, resulting in multiple edge points and multiple planar points. Multiple strong edge features are selected from the multiple edge points, and multiple strong planar features are selected from the multiple planar points. For each edge point and each planar point in each frame of the initial point cloud, each strong edge feature in the edge point and each strong planar feature in the planar point are sequentially traversed. In the edge points and planar points of the previous frame of the initial point cloud, a search is performed to find features that match each strong edge feature in the current frame of the initial point cloud. Edge points with a distance less than a preset distance are fitted into feature lines; plane points with a distance less than a preset distance from each strong plane feature in the initial point cloud of the current frame are searched and fitted into feature surfaces; residual blocks are constructed using the distance constraints of feature lines and feature surfaces, and semantic confidence scores are set for the residual blocks through the semantic labels of feature lines or feature surfaces to adaptively adjust the distance from each strong edge feature to the feature line and the distance from each strong plane feature to the feature surface, calculate the confidence of the residuals, and perform feature matching based on the confidence of the residuals to obtain the pose; compared with the existing technology, edge features and plane features are extracted through curvature relationship and semantic segmentation labels, which reduces the error in feature extraction, and semantic confidence scores are introduced on the basis of traditional registration algorithms to improve the accuracy of lidar ranging.

[0045] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0046] Figure 1This is a flowchart illustrating an embodiment of the present invention;

[0047] Figure 2 This is the initial point cloud image collected in an embodiment of the present invention. Detailed Implementation

[0048] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0050] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0051] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0052] This invention addresses existing problems by providing a radar odometry method and related equipment that integrates semantic features.

[0053] like Figure 1 As shown, embodiments of the present invention provide a radar odometry method that fuses semantic features, including:

[0054] Step 1: Collect multiple frames of initial point cloud data of the target environment using a lidar sensor;

[0055] Step 2: Project the initial point cloud of each frame to obtain the distance image of the initial point cloud of each frame, and input the distance image into the fully convolutional network for semantic segmentation to obtain the semantic segmentation label of the initial point cloud of each frame.

[0056] Step 3: Using semantic segmentation labels as constraints, perform feature differentiation on the initial point cloud of each frame according to the curvature threshold to obtain multiple edge points and multiple planar points, and select multiple strong edge features from the multiple edge points and multiple strong planar features from the multiple planar points.

[0057] Step 4: For each edge point in the initial point cloud of each frame, traverse each strong edge feature in the edge points in turn, search for edge points in the edge points of the previous frame initial point cloud that are less than a preset distance from each strong edge feature in the current frame initial point cloud and fit them into feature lines.

[0058] Step 5: For each plane point in the initial point cloud of each frame, traverse each strong plane feature in the plane point in turn, search for plane points in the plane points of the previous frame initial point cloud whose distance to each strong plane feature in the current frame initial point cloud is less than a preset distance, and fit them into a feature surface.

[0059] Step 6: Construct residual blocks using the distance constraints of feature lines and feature surfaces, and set semantic confidence scores for the residual blocks using the semantic labels of feature lines or feature surfaces;

[0060] Step 7: Adaptively adjust the distance from each strong edge feature to the feature line and the distance from each strong plane feature to the feature surface based on the semantic confidence score, calculate the confidence of the residual, and perform feature matching based on the confidence of the residual to obtain the pose.

[0061] Specifically, the steps preceding step 2 also include:

[0062] Point cloud filtering is performed on the initial point cloud of multiple frames to remove invalid points, resulting in the filtered point cloud of multiple frames.

[0063] The distance image is obtained by projecting the filtered point cloud from multiple frames.

[0064] In this embodiment of the invention, multiple frames of initial point cloud data of the target environment are collected by a lidar sensor. There are usually some unstable points and invalid points that are too close or too far from the radar coordinate system. These points do not represent the surrounding environment, but rather the device on which the lidar is mounted, such as the surface of a vehicle body. These invalid points not only affect the selection of corresponding points, but may also increase the burden of semantic segmentation. Therefore, it is necessary to perform point cloud filtering on the multiple frames of initial point cloud data to remove invalid points, points within 0.15m of the origin of the radar coordinate system, and points that are too far away based on the lidar scanID.

[0065] Specifically, the distance image is obtained by projecting multiple frames of initial point cloud data, including:

[0066] The image coordinates are obtained by spherically projecting the initial point clouds of multiple frames according to the following formula:

[0067]

[0068] Among them, among them, Represents the pixel coordinates of the image. The Euclidean distance is the initial point cloud to the origin of the lidar sensor. This represents the vertical viewing angle range of the radar, indicating the vertical viewing angle range of the radar. , This indicates the downward field of view range in the vertical direction of the radar. For the height of the 2D ranging image, Represents the coordinates of a 3D point cloud. The width of the 2D ranging image;

[0069] The distance image is obtained based on the image coordinates.

[0070] like Figure 2 As shown in the embodiment of the present invention, the initial point cloud collected in step 1 is disordered and cannot be directly used with deep learning networks. Current 3D LiDAR is usually 16 or 32 scans, and a large amount of point cloud data can be obtained in one frame, which is relatively complicated to process. In order to solve the above problems, the initial point cloud is first projected onto the image coordinates through a spherical surface, and then the distance image is obtained according to the image coordinates.

[0071] Specifically, the distance image is input into a fully convolutional network for semantic segmentation to obtain semantic segmentation labels for the initial point cloud of each frame, including:

[0072] The distance image is input into a fully convolutional network for semantic segmentation to obtain semantic labels for the pixels in the distance image. The fully convolutional network includes an hourglass encoder and a decoder.

[0073] By mapping the initial point cloud to the pixels of the distance image, semantic labels are assigned to the initial point cloud, thus obtaining the semantic segmentation label for each frame of the initial point cloud.

[0074] In this embodiment of the invention, the fully convolutional network extracts the Darknet deep learning framework from the YOLOv3 network. During training, stochastic gradient descent and weighted cross-entropy loss function are used to optimize the end-to-end process, as shown in the following formula:

[0075]

[0076] in, Represents frequency The weight of the inverse logarithm c class is used to penalize the weight. This represents the softmax function on the unbounded module, which is the last layer of the inference process.

[0077] Because the vertical method of the distance image stores the scanID information of the laser radar, which is much more important than the points in the horizontal direction, in order to prevent the loss of vertical information, upsampling and downsampling can only be performed in the horizontal direction.

[0078] Then, the semantic labels of the pixels in the semantically segmented distance image are assigned to all point clouds within that pixel, so that all point clouds can be assigned a label. It should be noted that there are multiple actual initial point clouds corresponding to the pixels in the semantically segmented distance image, and the corresponding semantic segmentation labels need to be found by using the correspondence between the initial point clouds and 2D pixels.

[0079] Specifically, step 3 includes:

[0080] The curvature of the initial point cloud in each frame is:

[0081]

[0082] in, This represents an initial point cloud frame. , , They represent the first Initial point cloud of frame, first Initial point cloud of frame, , , These are two initial point cloud frames acquired from the same wiring harness. Indicates the first The 3D coordinates of the initial point cloud in frame c, where c represents the first frame. The curvature of the point cloud at the beginning of the frame. Indicates the first Semantic segmentation labels for the initial point cloud of the frame;

[0083] Set a curvature threshold a, and perform feature differentiation on the initial point cloud of each frame based on the curvature threshold a. Points in the initial point cloud of each frame with curvature C greater than the curvature threshold a are taken as edge points, and points in the initial point cloud of each frame with curvature C less than the curvature threshold a are taken as plane points.

[0084] The initial point cloud in each frame, which is collected by the same wire harness, is divided into 6 parts;

[0085] For each part of the initial point cloud, with semantic segmentation labels as constraints, Ne strong edge features are selected from the edge points of the initial point cloud, and Np strong planar features are selected from the planar points of each part.

[0086] Based on determining planar features through the curvature relationship of point clouds, semantic segmentation labels can provide a strong constraint. For the same scanID, this embodiment of the invention uses point... The curvature is fitted using 5 points on each of the left and right adjacent points, and the values ​​of the points among 10 adjacent points are recorded. The number of points with the same semantic label, u, is expressed as follows:

[0087]

[0088] in, This represents the semantic threshold for determining whether a point is a planar feature point.

[0089] In this embodiment of the invention, there are two types of features that cannot be observed stably: the first type is features that are nearly parallel to the scanning lines of the mechanism, and the second type is features that are scanned at one end and are blocked by another plane. Therefore, these two types of features are unstable and need to be removed.

[0090] Specifically, feature matching is based on point cloud registration algorithms, where the nearest point iterative ICP is a classic local point cloud registration algorithm. It takes the closest point in the source point cloud and the nearest point in the target point cloud as the hypothetical corresponding point. Iterates using the least squares method until the error function value is less than a set threshold, at which point the iteration terminates and outputs R and t, as shown in the following expression:

[0091]

[0092] in, t and t represent rotation and translation, respectively. However, the ICP algorithm at this time only considers the error function of the distance between points and lacks the use of point cloud structure information. Therefore, in order to improve accuracy and avoid getting trapped in local optima, the self-motion of the estimated lidar is estimated by using point lines and point surfaces to construct error functions.

[0093] Specifically, step 4 includes:

[0094] For each edge point in the initial point cloud of each frame, each strong edge feature in the edge points is traversed sequentially. Using a K-tree, multiple edge points in the edge points of the previous frame's initial point cloud that are less than a preset distance from each strong edge feature in the current frame's initial point cloud are searched. Then, a feature line is fitted using principal component analysis.

[0095] Specifically, step 5 includes:

[0096] For each planar point in the initial point cloud of each frame, each strong planar feature in the planar point is traversed sequentially. Using a K-tree, planar points in the planar points of the previous frame's initial point cloud that are less than a preset distance from each strong planar feature in the current frame's initial point cloud are searched. Then, a feature surface is fitted by principal component analysis.

[0097] Specifically, steps 6 and 7 include:

[0098] Residual blocks are constructed using distance constraints between feature lines and feature surfaces, and semantic confidence scores are set for the residual blocks using semantic labels of feature lines or feature surfaces.

[0099] The distance from each strong edge feature to the feature line and the distance from each strong plane feature to the feature surface are adaptively adjusted based on the semantic confidence score. The confidence of the residuals is calculated, and feature matching is performed based on the confidence of the residuals to obtain the pose.

[0100] In this embodiment of the invention, the edge features of the previous frame are referred to as Ne, and the planar features as Np; the total edge features of the previous frame are... The overall planar features are For all strong feature points e i p i (belonging to Ne, Np) via a K-dimensional tree in , Search for the N nearest points and fit them to characteristic lines using principal component analysis (PCA). and characteristic surface p Feature matching is achieved by minimizing point e. i to the feature line distance With p i To feature surface p distance Solve the pose of the residual block.

[0101] However, since the extent to which the fitted feature lines and feature surfaces correspond to the geometric relationships in real space is unknown, a semantic confidence score is set based on semantic features to adaptively adjust the confidence of the fitted feature line and feature surface residuals; the semantic confidence score is:

[0102]

[0103] in, The semantic segmentation label used to determine strong edge features or strong planar features is the same as the kth point among the N nearest edge points searched. A value of 1 indicates the same label, and a value of 0 indicates a different label. Therefore, the final residual formula given in this embodiment of the invention is:

[0104]

[0105] in, , These are the semantic confidence scores for point-line and point-surface residuals, respectively. , These are the point-to-line distance and point-to-surface distance corresponding to the i-th feature point, respectively.

[0106] The following table compares the extraction of edge and planar features using curvature relationships versus using semantic information to assist in the extraction of edge and planar features, as presented in this invention:

[0107]

[0108] The table above clearly shows that the number of feature points extracted with the aid of semantic information is significantly less than the number of feature points extracted with the aid of curvature relationship.

[0109] This invention employs a lidar sensor to acquire multiple frames of initial point clouds of the target environment, projects them to obtain distance images, and inputs these distance images into a fully convolutional network for semantic segmentation, obtaining semantic segmentation labels for each frame of point cloud. Using these semantic segmentation labels as constraints, each frame of initial point cloud is feature-distinguished based on a curvature threshold, resulting in multiple edge points and multiple planar points. Multiple strong edge features are selected from the multiple edge points, and multiple strong planar features are selected from the multiple planar points. For each edge point and each planar point in each frame of initial point cloud, each strong edge feature in the edge point and each strong planar feature in the planar point are sequentially traversed. In the edge points and planar points of the previous frame of initial point cloud, the element with the smallest distance to each strong edge feature in the current frame of initial point cloud is searched. Edge points at preset distances are fitted into feature lines; plane points whose distance to each strong plane feature in the initial point cloud of the current frame is less than a preset distance are searched and fitted into feature surfaces; residual blocks are constructed using distance constraints of feature lines and feature surfaces, and semantic confidence scores are set for the residual blocks through semantic labels of feature lines or feature surfaces to adaptively adjust the distance from each strong edge feature to the feature line and the distance from each strong plane feature to the feature surface, calculate the confidence of the residuals, and perform feature matching based on the confidence of the residuals to obtain the pose; compared with existing technologies, edge features and plane features are extracted through curvature relationships and semantic segmentation labels, reducing the error in feature extraction, and semantic confidence scores are introduced on the basis of traditional registration algorithms to improve the accuracy of lidar ranging.

[0110] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a radar odometry method that fuses semantic features.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the methods described above by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0112] This invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a radar odometry method that fuses semantic features.

[0113] It should be noted that the terminal device can be a mobile phone, tablet computer, laptop computer, Ultra-mobile Personal Computer (UMPC), netbook, Personal Digital Assistant (PDA), etc. For example, the terminal device can be a station (ST) in a WLAN, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, PDA, handheld device with wireless communication capabilities, computing device or other processing device connected to a wireless modem, computer, laptop computer, handheld communication device, handheld computing device, satellite wireless device, etc. The embodiments of this invention do not limit the specific type of terminal device.

[0114] The processor referred to can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0115] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory may include both internal and external storage units of the terminal device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0116] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0117] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A radar odometry method incorporating semantic features, characterized in that, include: Step 1: Collect multiple frames of initial point cloud data of the target environment using a lidar sensor; Step 2: Project the initial point cloud of each frame to obtain the distance image of the initial point cloud of each frame, and input the distance image into a fully convolutional network for semantic segmentation to obtain the semantic segmentation label of the initial point cloud of each frame; Step 3: Using the semantic segmentation label as a constraint, perform feature differentiation on the initial point cloud of each frame according to the curvature threshold to obtain multiple edge points and multiple planar points, and select multiple strong edge features from the multiple edge points and multiple strong planar features from the multiple planar points; Step 4: For each edge point in the initial point cloud of each frame, sequentially traverse each strong edge feature in the edge points, search for edge points in the edge points of the previous frame's initial point cloud that are less than a preset distance from each strong edge feature in the current frame's initial point cloud, and fit them into feature lines. Step 5: For each plane point in the initial point cloud of each frame, sequentially traverse each strong plane feature in the plane point, search for plane points in the plane points of the previous frame initial point cloud whose distance to each strong plane feature in the current frame initial point cloud is less than a preset distance, and fit them into a feature surface. Step 6: Construct residual blocks using the distance constraints between the feature lines and the feature surfaces, and set semantic confidence scores for the residual blocks using the semantic labels of the feature lines or the feature surfaces; Step 7: Adaptively adjust the distance from each strong edge feature to the feature line and the distance from each strong plane feature to the feature surface based on the semantic confidence score, calculate the confidence of the residual, and perform feature matching based on the confidence of the residual to obtain the pose.

2. The radar odometry method based on fused semantic features according to claim 1, characterized in that, Projecting multiple frames of the initial point cloud to obtain a distance image includes: The image coordinates are obtained by spherically projecting the initial point cloud from multiple frames according to the following formula: ； in, Represents the pixel coordinates of the image. The distance is the Euclidean distance from the initial point cloud to the origin of the lidar sensor. This represents the vertical upward field of view range of the radar, indicating the vertical field of view range of the radar. , This indicates the downward field of view range in the vertical direction of the radar. For the height of the 2D ranging image, Represents the coordinates of a 3D point cloud. The width of the 2D ranging image; The distance image is obtained based on the image coordinates.

3. The radar odometry method based on fused semantic features according to claim 2, characterized in that, The distance image is input into a fully convolutional network for semantic segmentation to obtain semantic segmentation labels for the initial point cloud of each frame, including: The distance image is input into a fully convolutional network for semantic segmentation to obtain semantic labels for the pixels of the distance image. The fully convolutional network includes an hourglass encoder and a decoder. By using the correspondence between the initial point cloud and the pixels of the distance image, the semantic label is assigned to the initial point cloud to obtain the semantic segmentation label of the initial point cloud for each frame.

4. The radar odometry method based on fused semantic features according to claim 3, characterized in that, Step 3 includes: The curvature of the initial point cloud in each frame is: ； in, This represents an initial point cloud frame. , , They represent the first Initial point cloud of frame, first Initial point cloud of frame , , These are two initial point cloud frames acquired from the same wiring harness. Indicates the first The 3D coordinates of the initial point cloud in frame c, where c represents the first frame. The curvature of the point cloud at the beginning of the frame. Indicates the first Semantic segmentation labels for the initial point cloud of the frame; Set a curvature threshold a, and perform feature differentiation on the initial point cloud of each frame according to the curvature threshold a. Points in the initial point cloud of each frame with a curvature C greater than the curvature threshold a are taken as edge points, and points in the initial point cloud of each frame with a curvature C less than the curvature threshold a are taken as plane points. The initial point cloud in each frame, which is collected by the same wire bundle, is divided into 6 parts; For each part of the initial point cloud, using the semantic segmentation label as a constraint, Ne strong edge features are selected from the edge points of the initial point cloud, and Np strong planar features are selected from the planar points of each part.

5. The radar odometry method based on fused semantic features according to claim 4, characterized in that, Step 4 includes: For each edge point in the initial point cloud of each frame, each strong edge feature in the edge points is traversed sequentially. Using a K-tree, multiple edge points in the edge points of the previous frame's initial point cloud that are less than a preset distance from each strong edge feature in the current frame's initial point cloud are searched and fitted into feature lines using principal component analysis.

6. The radar odometry method based on fused semantic features according to claim 4, characterized in that, Step 5 includes: For each planar point in the initial point cloud of each frame, each strong planar feature in the planar point is traversed sequentially. Using a K-tree, planar points in the planar points of the previous frame's initial point cloud that are less than a preset distance from each strong planar feature in the current frame's initial point cloud are searched and fitted into feature surfaces using principal component analysis.

7. The radar odometry method based on fused semantic features according to claim 4, characterized in that, The semantic confidence score is: ； in, The semantic segmentation label used to determine whether strong edge features or strong planar features are the same as the kth point among the N nearest edge points searched.

8. The radar odometry method based on fused semantic features according to claim 1, characterized in that, The steps preceding step 2 also include: Point cloud filtering is performed on the initial point cloud of multiple frames to remove invalid points, resulting in a filtered point cloud of multiple frames. The distance image is obtained by projecting the filtered point cloud from multiple frames.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the radar odometry method with fused semantic features as described in any one of claims 1 to 8.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the radar odometry method with fused semantic features as described in any one of claims 1 to 8.

Images