Self-built map method based on 4D millimeter wave imaging radar and ground landmark semantics

Abstract

The present invention relates to a method for realizing self-built map by 4D millimeter wave imaging radar and semantics of ground signs for memory commuting driving, the method comprising: obtaining full radar point cloud frame and timestamp for received radar point cloud; performing semantic frame screening and matching processing on the point cloud of acquired ground sign elements to obtain ground sign semantic frame point cloud; performing factor graph optimization processing by using self-vehicle sensor GNSS, inertial measurement unit IMU and wheel speed odometer to estimate the position of self-vehicle in real time; performing radar semantic mapping by using acquired radar point cloud frame, ground sign semantic frame and self-vehicle estimated posture queue to generate self-built map for current vehicle. The present invention also relates to a corresponding device, processor and storage medium thereof. The method, device, processor and storage medium thereof of the present invention are adopted to improve the stability and robustness of positioning based on self-built map during commuting driving, and bring comfortable intelligent driving experience to users.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to the field of commuting memory driving, and specifically refers to a method, device, processor and computer-readable storage medium thereof for realizing self-built maps for memory commuting driving through 4D millimeter wave imaging radar and ground sign semantics. Background Art

[0002] Memory commuting driving is an intelligent driving function that provides users with an autonomous driving function on a pre-memorized (preset) route. It is mainly suitable for high-frequency, fixed-route driving scenarios such as daily commuting. In this function, determining the real-time position of a moving vehicle in the environment, that is, vehicle positioning, is a key and essential technology.

[0003] At present, the positioning technology that matches lidar point cloud and high-precision map positioning elements is generally used in commuting memory driving. However, the hardware cost of lidar is relatively high, at least several thousand yuan, which makes it difficult to mass produce. On the other hand, high-precision maps have high production costs and long production cycles, and map updates are slow and lack freshness. Untimely map updates may lead to positioning errors, which in turn cause unfriendly driving experiences such as human takeover of vehicle control. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, device, processor and computer-readable storage medium thereof for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics to realize self-built map.

[0005] In order to achieve the above-mentioned purpose, the present invention proposes a low-cost, easy-to-mass-produce, all-weather 4D millimeter-wave imaging radar and ground sign semantics self-map construction method to generate a self-built map generated in the memory stage of the commuting process for positioning the prior map during autonomous driving. Compared with laser, this method has lower cost and is easier to mass produce for 4D millimeter-wave imaging radar; compared with high-precision maps, there are no problems such as freshness, complex production, and long production cycle, and updates only require data collection. Among them, the 4D millimeter-wave radar mainly covers the map construction of the environment above the ground around the vehicle, and the ground sign semantics constructs traffic signs such as lane lines, arrows, stop lines, etc. on the ground. The self-built map finally generated fully covers the ground of the vehicle and the surrounding environment information on the ground, providing a lightweight map with comprehensive information for positioning.

[0006] The main feature of the hybrid strategy-based radar-visual fusion target perception method is that the method comprises the following steps:

[0007] (1) Obtain the full radar point cloud frame and timestamp for the radar point cloud received by the 4D millimeter wave radar;

[0008] (2) Perform semantic frame screening and matching on the acquired point cloud of ground sign elements to obtain the corresponding ground sign semantic frame point cloud;

[0009] (3) Using the ego vehicle’s sensors, GNSS, IMU, and wheel speed odometer, the factor graph is optimized to estimate the ego vehicle’s position in real time.

[0010] (4) Using the acquired radar point cloud frames, ground landmark semantic frames, and the estimated pose queue of the ego vehicle, radar semantic mapping is performed to generate a self-built map for the current vehicle.

[0011] Preferably, the step (1) is specifically as follows:

[0012] The radar frame construction module receives radar point clouds from the main radar in front of the vehicle, the left front corner radar, the right front corner radar, the left rear corner radar, the right rear corner radar, and the main radar directly behind the vehicle. By synchronizing the timestamps of the six radars and then transforming the extrinsic parameters between the 4D millimeter-wave radar and the vehicle coordinates, the 4D millimeter-wave radar point cloud is converted to the vehicle coordinates to obtain the full radar point cloud frame and timestamp after the fusion of the six 4D millimeter-wave radars.

[0013] Preferably, the step (1) performs radar time synchronization processing in the following manner:

[0014] Assume that the currently acquired vehicle point cloud information is ,in , , , n is The number of radar points in the It means the jth radar is at time i, and the timestamp is , the point cloud information is , By n Point composition;

[0015] At time i, within a radar cycle, all radar timestamps obtained Find the maximum radar timestamp in , k is the radar number corresponding to the maximum timestamp, with the maximum radar timestamp As a benchmark, synchronize other radar timestamps to Then, the time difference is used to compensate the position information of other radars, as follows:

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] ;

[0022] in, Represents each radar timestamp and the maximum timestamp The time difference between and are the angular velocity and linear velocity of the vehicle chassis at time i, and For The motion compensation amount of the radar point in the x and y directions during the time period, for Rotated by angle in time, and is the jth radar after time synchronization alignment For each point Position coordinates in the x,y direction.

[0023] Preferably, the step (1) further comprises performing radar external parameter change processing:

[0024] Assume that the transformation between each radar and the vehicle coordinates is , then the external parameter transformation is transformed as follows:

[0025] ;

[0026] is the radar point cloud after external parameter conversion of the j-th radar at time i;

[0027] After the above processing, the radar frame point cloud is ,in is the maximum radar timestamp ,

[0028] ;

[0029] Right now is the radar point obtained after the external parameters of the six radars are transformed at time i The result after splicing.

[0030] Preferably, the step (2) specifically includes:

[0031] Semantic screening: semantic frame point cloud of input Each point in , make the following judgment:

[0032] ;

[0033] is the score threshold for judging whether a semantic point is selected, for The detection confidence of the mth point in is , then point Select, otherwise from Delete from;

[0034] Semantic matching: If the current time is , the previous moment is , the speed and angular velocity obtained from the vehicle chassis at the current moment are and , the semantic point cloud at the current moment is , the previous moment is , then the relative position transformation of the current moment relative to the previous moment for:

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] but

[0040] ;

[0041] The semantic point cloud of the previous moment Translated to the current moment, we get :

[0042] ;

[0043] in, Indicates the time difference between the current moment and the previous moment. for The angle rotated in time, and for The motion compensation amount of the semantic frame in the x and y directions within the time period;

[0044] Semantic Point Cloud Each point exist The nearest neighbor search is performed in , and the threshold is set to 0.05m. Found in The neighboring points of Keep in Otherwise, from Delete from the above; thereby obtaining the semantic frame point cloud of the ground landmark ,in Timestamp for semantic frame input , It is the semantic point cloud location information after semantic screening and semantic matching.

[0045] Preferably, the step (3) specifically includes the following processing:

[0046] GNSS absolute constraint: Assume that the GNSS observation value at time i is , the estimated posture of the vehicle at time i is , is the rotation information of the vehicle at time i, is the position of the vehicle at time i, then the following constraints must be met at time i:

[0047] ;

[0048] in, For subject to, it means being bound;

[0049] IMU constraint: Assuming the current time is i, the acceleration input by the IMU is , the angular velocity is , but At time i, the following constraints need to be met:

[0050] ;

[0051] ;

[0052] in To find the second-order derivative of the translation vector, To derive the rotation matrix;

[0053] Wheel speed odometer position constraint: Assume that the current time i is relative to the previous time i-1. ,but At time i, constraints are processed as follows:

[0054] ;

[0055] Perform factor graph optimization, the error of the factor graph for

[0056] ;

[0057] in, is the observed position of the ego-vehicle;

[0058] The error equation is:

[0059] ;

[0060] The information matrix of the error is then used to solve the optimal solution that satisfies the above constraints using the LM method. , get the estimate of the current position of the vehicle and the time corresponding to each position , enter the self-vehicle posture queue middle.

[0061] Preferably, the step (4) performs key frame time synchronization in the following manner:

[0062] From the self-vehicle pose queue In the above example, find the radar frame point cloud and semantic frame point cloud The most recent pose in time, once found, the old pose time is changed from Delete it to get the radar key frame And semantic keyframes ,in:

[0063] ;

[0064] .

[0065] and They represent the point clouds of the radar point cloud frame and the semantic point cloud frame in the vehicle coordinates at time i respectively.

[0066] Preferably, the method described above performs radar semantic mapping in the following manner:

[0067] The radar key frame obtained after time synchronization and semantic keyframes Stitching to get a self-built map :

[0068] ;

[0069] in, is the number of frames after the radar key frame time and posture synchronization, It is the frame number after the time and position synchronization of the ground landmark semantic frame.

[0070] The main features of the system for self-building maps for memory commuting driving through 4D millimeter wave imaging radar and ground sign semantics using the above-mentioned method are that the system includes:

[0071] The radar frame construction module is used to perform radar time synchronization and radar external parameter change processing on the received 6 millimeter-wave radar point clouds, so as to obtain the full radar point cloud frame and timestamp after the fusion of 6 radars;

[0072] The semantic frame construction module is used to perform semantic screening and semantic matching on the input ground sign semantics to obtain the 3D position information of the semantic elements in the vehicle coordinates;

[0073] The pose generation module is used to optimize the factor graph using the ego vehicle sensors GNSS, inertial measurement unit IMU, and wheel speed odometer to estimate the ego vehicle position information in real time; and

[0074] The self-built map generation module is connected to the radar frame construction module, the semantic frame construction module and the pose generation module, and is used for performing key frame time synchronization and radar semantic mapping processing to generate a self-built map.

[0075] The device for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics has the following main features: the device comprises:

[0076] a processor configured to execute computer-executable instructions;

[0077] A memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for building a self-map for memorizing commuting driving through 4D millimeter wave imaging radar and ground sign semantics are implemented.

[0078] The processor for realizing self-built maps for memorizing commuting driving through 4D millimeter-wave imaging radar and semantics of ground signs has the main feature that the processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for realizing self-built maps for memorizing commuting driving through 4D millimeter-wave imaging radar and semantics of ground signs are implemented.

[0079] The computer-readable storage medium has the main feature of storing a computer program thereon, which can be executed by a processor to implement the steps of the above-mentioned method for memorizing commuting driving through 4D millimeter-wave imaging radar and ground sign semantics to achieve self-built maps.

[0080] The method, device, processor and computer-readable storage medium of the present invention for realizing self-built maps for memory commuting driving through 4D millimeter-wave imaging radar and ground sign semantics overcome the problems of high cost and difficulty in mass production of laser radar and complex mapping cycle and slow update using high-precision maps. The self-built map generated by the method also includes point cloud information of ground semantic signs and point cloud information of 4D millimeter-wave imaging radar. The combination greatly improves the stability and robustness of positioning based on self-built maps during commuting driving, bringing users a comfortable intelligent driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a flow chart of the method of the present invention for realizing self-building maps for memory commuting driving through 4D millimeter wave imaging radar and ground sign semantics. DETAILED DESCRIPTION

[0082] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0083] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

[0084] First, some technical abbreviations used in this technical solution are supplemented:

[0085] radar: 4D millimeter wave imaging radar;

[0086] Pose: pose, including x, y, z position information and roll, pitch, yaw attitude information;

[0087] GNSS: Global Navigation Satellite System;

[0088] IMU: Inertial Measurement Unit, inertial measurement unit;

[0089] Self-built map: Relatively speaking, it is a map built by humans and generated by the driving vehicle itself.

[0090] See also Figure 1 As shown, the method for self-building a map for memory commuting driving through 4D millimeter wave imaging radar and ground sign semantics, wherein the method comprises the following steps:

[0091] (1) Obtain the full radar point cloud frame and timestamp for the radar point cloud received by the 4D millimeter wave radar;

[0092] (2) Perform semantic frame screening and matching on the acquired point cloud of ground sign elements to obtain the corresponding ground sign semantic frame point cloud;

[0093] (3) Using the ego vehicle’s sensors, GNSS, IMU, and wheel speed odometer, the factor graph is optimized to estimate the ego vehicle’s position in real time.

[0094] (4) Using the acquired radar point cloud frames, ground landmark semantic frames, and the estimated pose queue of the ego vehicle, radar semantic mapping is performed to generate a self-built map for the current vehicle.

[0095] As a preferred embodiment of the present invention, the step (1) is specifically as follows:

[0096] The radar frame construction module receives radar point clouds from the main radar in front of the vehicle, the left front corner radar, the right front corner radar, the left rear corner radar, the right rear corner radar, and the main radar directly behind the vehicle. By synchronizing the timestamps of the six radars and then transforming the extrinsic parameters between the 4D millimeter-wave radar and the vehicle coordinates, the 4D millimeter-wave radar point cloud is converted to the vehicle coordinates to obtain the full radar point cloud frame and timestamp after the fusion of the six 4D millimeter-wave radars.

[0097] As a preferred embodiment of the present invention, the step (1) performs radar time synchronization processing in the following manner:

[0098] Assume that the currently acquired vehicle point cloud information is ,in , , , n is The number of radar points in the It means the jth radar is at time i, and the timestamp is , the point cloud information is , By n Point composition;

[0099] At time i, within a radar cycle, all radar timestamps obtained Find the maximum radar timestamp in , k is the radar number corresponding to the maximum timestamp, with the maximum radar timestamp As a benchmark, synchronize other radar timestamps to Then, the time difference is used to compensate the position information of other radars, as follows:

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] in, Represents each radar timestamp and the maximum timestamp The time difference between and are the angular velocity and linear velocity of the vehicle chassis at time i, and For The motion compensation amount of the radar point in the x and y directions during the time period, for Rotated by angle in time, and is the jth radar after time synchronization alignment For each point Position coordinates in the x,y direction.

[0107] As a preferred embodiment of the present invention, the step (1) further includes performing radar external parameter change processing:

[0108] Assume that the transformation between each radar and the vehicle coordinates is , then the external parameter transformation is transformed as follows:

[0109] ;

[0110] is the radar point cloud after external parameter conversion of the j-th radar at time i;

[0111] After the above processing, the radar frame point cloud is ,in is the maximum radar timestamp ,

[0112] ;

[0113] Right now is the radar point obtained after the external parameters of the six radars are transformed at time i The result after splicing.

[0114] As a preferred embodiment of the present invention, the step (2) specifically includes:

[0115] Semantic screening: semantic frame point cloud of input Each point in , make the following judgment:

[0116] ;

[0117] is the score threshold for judging whether a semantic point is selected, for The detection confidence of the mth point in is , then point Select, otherwise from Delete from;

[0118] Semantic matching: If the current time is , the previous moment is , the speed and angular velocity obtained from the vehicle chassis at the current moment are and , the semantic point cloud at the current moment is , the previous moment is , then the relative position transformation of the current moment relative to the previous moment for:

[0119] ;

[0120] ;

[0121] ;

[0122] ;

[0123] but

[0124] ;

[0125] The semantic point cloud of the previous moment Translated to the current moment, we get :

[0126] ;

[0127] in, Indicates the time difference between the current moment and the previous moment. for The angle rotated in time, and for The motion compensation amount of the semantic frame in the x and y directions within the time period;

[0128] Semantic Point Cloud Each point exist The nearest neighbor search is performed in , and the threshold is set to 0.05m. Found in The neighboring points of Keep in Otherwise, from Delete from the above; thereby obtaining the semantic frame point cloud of the ground landmark ,in Timestamp for semantic frame input , It is the semantic point cloud location information after semantic screening and semantic matching.

[0129] As a preferred embodiment of the present invention, the step (3) specifically includes the following processing:

[0130] GNSS absolute constraint: Assume that the GNSS observation value at time i is , the estimated posture of the vehicle at time i is , is the rotation information of the vehicle at time i, is the position of the vehicle at time i, then the following constraints must be met at time i:

[0131] ;

[0132] in, For subject to, it means being bound;

[0133] IMU constraint: Assuming the current time is i, the acceleration input by the IMU is , the angular velocity is , but At time i, the following constraints need to be met:

[0134] ;

[0135] ;

[0136] in To find the second-order derivative of the translation vector, To derive the rotation matrix;

[0137] Wheel speed odometer position constraint: Assume that the current time i is relative to the previous time i-1. ,but At time i, constraints are processed as follows:

[0138] ;

[0139] Perform factor graph optimization, the error of the factor graph for

[0140] ;

[0141] in, is the observed position of the ego-vehicle;

[0142] The error equation is:

[0143] ;

[0144] The information matrix of the error is then used to solve the optimal solution that satisfies the above constraints using the LM method. , get the estimate of the current position of the vehicle and the time corresponding to each position , enter the self-vehicle posture queue middle.

[0145] As a preferred embodiment of the present invention, the step (4) performs key frame time synchronization in the following manner:

[0146] From the self-vehicle pose queue In the above example, find the radar frame point cloud and semantic frame point cloud The most recent pose in time, once found, the old pose time is changed from Delete it to get the radar key frame And semantic keyframes ,in:

[0147] ;

[0148] .

[0149] and They represent the point clouds of the radar point cloud frame and the semantic point cloud frame in the vehicle coordinates at time i respectively.

[0150] As a preferred embodiment of the present invention, the method described herein performs radar semantic mapping in the following manner:

[0151] The radar key frame obtained after time synchronization and semantic keyframes Stitching to get a self-built map :

[0152] ;

[0153] in, is the number of frames after the radar key frame time and posture synchronization, It is the frame number after the time and position synchronization of the ground landmark semantic frame.

[0154] The various components of this technical solution will be further described in detail below:

[0155] 1. Radar frame construction module

[0156] The radar frame construction module receives 6 radar point clouds, which are from the main radar in front of the vehicle, the left front corner radar, the right front corner radar, the left rear corner radar, the right rear corner radar, and the main radar directly behind the vehicle. After synchronizing the timestamps of the 6 radars, the radar point cloud is converted to the vehicle coordinates according to the external parameter transformation between the radar and the vehicle coordinates, and the full radar point cloud frame and timestamp after the fusion of the 6 radars are obtained.

[0157] The specific processing flow is as follows: Assume The number of radar points in the It means the jth radar is at time i, and the timestamp is , the point cloud information is , It is composed of n Point composition.

[0158] a) Radar time synchronization

[0159] At time i, within a radar cycle of 100ms, from all radar timestamps arriving at the system Find the maximum timestamp in , k is the radar number corresponding to the maximum timestamp, and the radar timestamp that arrives at the system latest As a benchmark, synchronize other radar timestamps to , using the time difference to compensate for other radars of The position information, compensation formula is as follows:

[0160] ;

[0161] ;

[0162] ;

[0163] ;

[0164] ;

[0165] ;

[0166] in, Represents each radar timestamp and the maximum timestamp The time difference between and are the angular velocity and linear velocity of the vehicle chassis at time i, and For The motion compensation amount of the radar point in the x and y directions during the time period, for Rotated by angle in time, and is the jth radar after time synchronization alignment For each point Position coordinates in the x,y direction.

[0167] b) Radar extrinsic parameter transformation

[0168] Since each radar point in a) It is expressed in the radar's own coordinate system, and is uniformly transformed into the vehicle coordinate system with the center of the vehicle's rear axle as the origin. Assume that the transformation between each radar and the vehicle coordinate system is , then the external parameter transformation is transformed according to the following formula:

[0169] ;

[0170] is the radar point cloud after extrinsic conversion of the j-th radar at time i.

[0171] After processing by the above two modules, the radar frame point cloud ,in For the above ;

[0172] ;

[0173] Right now is the radar point after the external parameters of the six radars are transformed at time i The result after splicing.

[0174] 2. Semantic frame construction module

[0175] The semantic input of ground sign is the point cloud of ground sign elements (lane lines, arrows, stop lines, etc.). The semantic point cloud comes from the image detection result of the 800w camera of the self-vehicle front view, not the result of the instance segmentation of the bird's-eye view spliced ​​from the original fisheye image. That is, the pixels of the ground elements detected within a certain range of the image are sampled every 10 pixels. The ground elements have no semantic labels and are uniformly regarded as point clouds. Then the pixels are converted to the self-vehicle coordinates using the internal and external references of the camera to obtain the 3D position of the semantic elements in the self-vehicle coordinates. The number of semantic point clouds in Points The position coordinates of the x and y axes in the vehicle coordinates, for The confidence score of The timestamp of the semantic frame at time i is , the point cloud location information is .

[0176] Since the confidence score of the ground landmark semantic point Different from the above, some points with relatively low scores are filtered out through the semantic screening module. At the same time, in order to ensure that the semantics of ground landmarks exist stably in the previous and next semantic frames, the semantic matching module of the previous and next frames retains the stable semantic points as the current semantic frame point cloud and outputs them to the self-built map module.

[0177] a) Semantic screening: input semantic frame point cloud Each point in , make the following judgment

[0178] ;

[0179] It is the score threshold for judging whether a semantic point is selected. for The detection confidence of the mth point in is , then point Select, otherwise from Delete it.

[0180] b) Semantic matching: If the current moment is , the previous moment is , the speed and angular velocity obtained from the vehicle chassis at the current moment are and , the semantic point cloud at the current moment is , the previous moment is , then the relative position transformation of the current moment relative to the previous moment ;

[0181] ;

[0182] ;

[0183] ;

[0184] ;

[0185] but

[0186] ;

[0187] The semantic point cloud of the previous moment Translated to the current moment, we get ;

[0188] ;

[0189] right Each point exist The nearest neighbor search is performed in , and the threshold is set to 0.05m. Found in The neighboring points of Keep in Otherwise, from After the above semantic screening and semantic matching modules, the semantic frame point cloud of the ground landmark is obtained. , Timestamp for semantic frame input Remain unchanged, It is the semantic point cloud location information after semantic screening and semantic matching.

[0190] 3. Pose generation module

[0191] The vehicle position is estimated in real time using the vehicle sensors GNSS, IMU, and wheel speed odometer. GNSS provides global position constraints for the vehicle, IMU provides constraints on the acceleration and angular velocity of the vehicle, and wheel speed odometer provides constraints on relative position. The three constraints are used to construct a factor graph and perform nonlinear optimization to solve the current position of the vehicle. , yes The real rotation matrix of yes A real translation vector of .

[0192] a) GNSS absolute constraint: Assume that the GNSS observation value at time i is , the estimated posture of the vehicle at time i is , is the rotation information of the vehicle at time i, is the position of the vehicle at time i, then the following constraints must be met at time i:

[0193] ;

[0194] b) IMU constraint: Assume that the current time is i, and the acceleration input by the IMU is , the angular velocity is , but At time i, the following constraints need to be met:

[0195] ;

[0196] ;

[0197] in To find the second-order derivative of the translation vector, To find the derivative of the rotation matrix.

[0198] c) Wheel speed odometer position constraint: If the current time i is relative to the previous time i-1, the vehicle moves ,but At time i, the following position constraints need to be met:

[0199] ;

[0200] d) Factor graph optimization, factor graph error for

[0201] ;

[0202] in, is the position observed by the vehicle, mainly measured by GNSS. Error equation

[0203] ;

[0204] is the information matrix of the error, generally a unit diagonal matrix of 10-6.

[0205] Use the LM method to find the optimal solution that satisfies the above constraints. . Get the current position estimate of the vehicle and the time corresponding to each position , enter the self-vehicle posture queue middle.

[0206] 4. Self-built graph module

[0207] Using radar point cloud frames and ground landmark semantic frame And the pose queue estimated by the ego vehicle Perform radar semantic mapping to generate a self-built map. This mainly includes two processes: key frame time synchronization and radar semantic mapping.

[0208] a) Keyframe time synchronization: from the pose queue estimated by the ego vehicle In the radar frame and semantic frames The closest pose in time. That is, for the radar frame time from Find in traversal and The most recent position , for the semantic frame time from Find in traversal and The most recent position , after finding, the older pose time is changed from Through time synchronization, radar key frames are obtained respectively And semantic keyframes ,in

[0209] ;

[0210] .

[0211] and They represent the point clouds of the radar point cloud frame and the semantic point cloud frame in the vehicle coordinates at time i respectively.

[0212] b) Radar semantic mapping: radar key frames obtained after time synchronization and semantic keyframes Stitching to get a self-built map :

[0213] ;

[0214] is the number of frames after the radar keyframe time and posture synchronization, It is the frame number after the time and position synchronization of the ground landmark semantic frame.

[0215] The device for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics, wherein the device comprises:

[0216] a processor configured to execute computer-executable instructions;

[0217] A memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for building a self-map for memorizing commuting driving through 4D millimeter wave imaging radar and ground sign semantics are implemented.

[0218] The processor for realizing self-built maps for memorizing commuting driving through 4D millimeter wave imaging radar and semantics of ground signs, wherein the processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for realizing self-built maps for memorizing commuting driving through 4D millimeter wave imaging radar and semantics of ground signs are implemented.

[0219] The computer-readable storage medium stores a computer program thereon, and the computer program can be executed by a processor to implement the steps of the above-mentioned method for memorizing commuting driving through 4D millimeter wave imaging radar and ground sign semantics to achieve self-built maps.

[0220] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0221] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0222] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0223] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0224] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0225] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0226] The method, device, processor and computer-readable storage medium of the present invention for realizing self-built maps for memory commuting driving through 4D millimeter-wave imaging radar and ground sign semantics overcome the problems of high cost and difficulty in mass production of laser radar and complex mapping cycle and slow update using high-precision maps. The self-built map generated by the method also includes point cloud information of ground semantic signs and point cloud information of 4D millimeter-wave imaging radar. The combination greatly improves the stability and robustness of positioning based on self-built maps during commuting driving, bringing users a comfortable intelligent driving experience.

[0227] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics, characterized in that: The method comprises the following steps: (1) Obtain the full radar point cloud frame and timestamp for the radar point cloud received by the 4D millimeter wave radar; (2) Perform semantic frame screening and matching on the acquired point cloud of ground sign elements to obtain the corresponding ground sign semantic frame point cloud; (3) Using the ego vehicle’s sensors, GNSS, IMU, and wheel speed odometer, the factor graph is optimized to estimate the ego vehicle’s position in real time. (4) Using the acquired radar point cloud frames, ground landmark semantic frames, and the estimated pose queue of the ego vehicle to perform radar semantic mapping, thereby generating a self-built map for the current vehicle; The step (4) performs key frame time synchronization in the following manner: From the ego vehicle pose queue In the above example, find the radar frame point cloud and semantic frame point cloud The most recent pose in time, where Estimated position of the vehicle at time i , is the rotation information of the vehicle at time i, is the position of the vehicle at time i, is the time corresponding to each posture, is the maximum radar timestamp , is the radar point obtained after the external parameter transformation of the six radars from the main radar in front of the vehicle, the left front corner radar, the right front corner radar, the left rear corner radar, the right rear corner radar, and the main radar directly behind the vehicle at time i The result after splicing, Timestamp for semantic frame input , It is the semantic point cloud location information after semantic screening and semantic matching; After finding a pose that meets the requirements, the old pose time is changed from Delete it to get the radar key frame And semantic keyframes ,in: ； ； and They represent the point clouds of the radar point cloud frame and the semantic point cloud frame in the ego-vehicle coordinates at time i respectively; The method described above performs radar semantic mapping in the following manner: The radar key frame obtained after time synchronization and semantic keyframes Stitching to get a self-built map : ； in, is the number of frames after the radar key frame time and posture synchronization, It is the frame number after the time and position synchronization of the ground landmark semantic frame.

2. The method for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics according to claim 1, characterized in that: The step (1) is specifically as follows: The radar frame construction module receives radar point clouds from the main radar in front of the vehicle, the left front corner radar, the right front corner radar, the left rear corner radar, the right rear corner radar, and the main radar directly behind the vehicle. By synchronizing the timestamps of the six radars and then transforming the extrinsic parameters between the 4D millimeter-wave radar and the vehicle coordinates, the 4D millimeter-wave radar point cloud is converted to the vehicle coordinates to obtain the full radar point cloud frame and timestamp after the fusion of the six 4D millimeter-wave radars.

3. The method for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics according to claim 2, characterized in that: The step (1) performs radar time synchronization processing in the following manner: Assume that the currently acquired vehicle point cloud information is ,in , , , n is The number of radar points in the It means the jth radar is at time i, and the timestamp is , the point cloud information is , By n Point composition; At time i, within a radar cycle, all radar timestamps obtained Find the maximum radar timestamp in , k The radar number corresponding to the maximum timestamp. As a benchmark, synchronize other radar timestamps to Then, the time difference is used to compensate the position information of other radars, as follows: ； ； ； ； ； ； in, Represents each radar timestamp and the maximum timestamp The time difference between and are the angular velocity and linear velocity of the vehicle chassis at time i, and For The motion compensation amount of the radar point in the x and y directions during the time period, for The angle rotated in time, and is the jth radar after time synchronization alignment For each point Position coordinates in the x,y direction.

4. The method for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics according to claim 3, characterized in that: The step (1) further includes performing radar external parameter change processing: Assume that the transformation between each radar and the vehicle coordinates is , then the external parameter transformation is transformed as follows: ； is the radar point cloud after external parameter conversion of the j-th radar at time i; After the above processing, the radar frame point cloud is ,in is the maximum radar timestamp , ； Right now is the radar point obtained after the external parameters of the six radars are transformed at time i The result after splicing.

5. The method for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics according to claim 4, characterized in that: The step (2) specifically includes: Semantic screening: semantic frame point cloud of input Each point in , make the following judgment: ； The score threshold for judging whether a semantic point is selected. for The detection confidence of the mth point in is , then point Select, otherwise from Delete from; Semantic matching: If the current time is , the previous moment is , the speed and angular velocity obtained from the vehicle chassis at the current moment are and , the semantic point cloud at the current moment is , the previous moment is , then the relative position transformation of the current moment relative to the previous moment for: ； ； ； ； but ； The semantic point cloud of the previous moment Translated to the current moment, we get : ； in, Indicates the time difference between the current moment and the previous moment. for The angle rotated in time, and for The motion compensation amount of the semantic frame in the x and y directions within the time period; Semantic Point Cloud Each point exist The nearest neighbor search is performed in , and the threshold is set to 0.05m. Found in The neighboring points of Keep in Otherwise, from Delete from the above; thereby obtaining the semantic frame point cloud of the ground landmark ,in Timestamp for semantic frame input , It is the semantic point cloud location information after semantic screening and semantic matching.

6. The method for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics according to claim 5, characterized in that: The step (3) specifically includes the following processing: GNSS absolute constraint: Assume that the GNSS observation value at time i is , the estimated posture of the vehicle at time i is , is the rotation information of the vehicle at time i, is the position of the vehicle at time i, then the following constraints must be met at time i: ； in, For subject to, it means being bound; IMU constraint: Assuming the current time is i, the acceleration input by the IMU is , the angular velocity is , but At time i, the following constraints need to be met: ； ； in To find the second-order derivative of the translation vector, To derive the rotation matrix; Wheel speed odometer position constraint: Assume that the current time i is relative to the previous time i-1. ,but At time i, constraints are processed as follows: ； Perform factor graph optimization, the error of the factor graph for ； in, is the observed position of the ego-vehicle; The error equation is: ； in, m is the number of all factors, The information matrix of the error is then used to solve the optimal solution that satisfies the above constraints using the LM method. , get the estimate of the current position of the vehicle and the time corresponding to each position , enter the self-vehicle posture queue middle.

7. A system for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics to realize self-building map using the method described in any one of claims 1 to 6, characterized in that: The system comprises: The radar frame construction module is used to perform radar time synchronization and radar external parameter change processing on the received 6 millimeter-wave radar point clouds, so as to obtain the full radar point cloud frame and timestamp after the fusion of 6 radars; The semantic frame construction module is used to perform semantic screening and semantic matching on the input ground sign semantics to obtain the 3D position information of the semantic elements in the vehicle coordinates; The pose generation module is used to optimize the factor graph using the ego vehicle sensors GNSS, inertial measurement unit IMU, and wheel speed odometer to estimate the ego vehicle position information in real time; and The self-built map generation module is connected to the radar frame construction module, the semantic frame construction module and the pose generation module, and is used for performing key frame time synchronization and radar semantic mapping processing to generate a self-built map.

8. A device for building a self-map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics, characterized in that: The device comprises: a processor configured to execute computer-executable instructions; A memory storing one or more computer executable instructions, wherein when the computer executable instructions are executed by the processor, the steps of the method for self-building a map for memorizing commuting driving by using 4D millimeter wave imaging radar and ground sign semantics as described in any one of claims 1 to 6 are implemented.

9. A processor for memorizing commuting driving and building a self-map through 4D millimeter wave imaging radar and ground sign semantics, characterized in that: The processor is configured to execute computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for self-building a map for memorizing commuting driving through 4D millimeter wave imaging radar and ground sign semantics as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method for self-building a map for memorizing commuting driving through 4D millimeter wave imaging radar and ground sign semantics as described in any one of claims 1 to 6.

Images