Multi-layer high-precision map generation method and device

Abstract

The application provides a multi-layer high-precision map generation method and device. A method for generating a multi-layer high-precision map comprises: receiving data from a plurality of sensors on a vehicle, the plurality of sensors comprising at least a millimeter wave radar and an ultrasonic radar; constructing a driving track of the vehicle according to data collected by the plurality of sensors; fusing millimeter wave point cloud data collected by the millimeter wave radar and ultrasonic point cloud data collected by the ultrasonic radar to generate a millimeter wave-ultrasonic information layer according to the driving track of the vehicle; and generating a high-precision map by using the millimeter wave-ultrasonic information layer.

Description

Technical Field

[0001] The present disclosure generally relates to the field of intelligent driving, and more particularly to a method and apparatus for generating multi-layer high-precision maps. Background Art

[0002] In recent years, as autonomous driving technology continues to mature, vehicles with autonomous driving capabilities are increasingly appearing in daily life. As an essential component of autonomous driving, the creation and updating of high-precision maps (HD maps) has become a key research topic within the industry.

[0003] Currently, sensors such as visual sensors, Global Navigation Satellite Systems (GNSS), and radar are commonly used to collect data and build maps. However, as vehicles drive, they encounter various environmental changes (for example, low light levels and weak network signals), and existing mapping solutions sometimes fall short. Summary of the Invention

[0004] To address the above technical problems in the prior art, this application provides a method for generating a multi-layer high-precision map, including:

[0005] receiving data from a plurality of sensors on the vehicle, the plurality of sensors including at least a millimeter-wave radar and an ultrasonic radar;

[0006] constructing a driving trajectory of the vehicle based on data collected by the multiple sensors;

[0007] According to the driving trajectory of the vehicle, the millimeter wave point cloud data collected by the millimeter wave radar and the ultrasonic point cloud data collected by the ultrasonic radar are fused to generate a millimeter wave-ultrasonic wave information layer; and

[0008] The millimeter wave-ultrasonic wave information layer is used to generate a high-precision map.

[0009] Optionally, fusing the millimeter wave point cloud data and the ultrasonic point cloud data includes:

[0010] For each trajectory point on the driving trajectory, determining a measurement value of the millimeter wave point cloud data and the ultrasonic point cloud data collected at the trajectory point; and

[0011] One of the millimeter wave point cloud data and the ultrasonic point cloud data is selected as the corresponding point cloud data at the trajectory point according to the measurement values ​​of the millimeter wave point cloud data and the ultrasonic point cloud data.

[0012] Optionally, the plurality of sensors include visual sensors, the method further comprising:

[0013] processing the image captured by the visual sensor to detect a first object and determine a category of the first object;

[0014] determining point cloud data matching the first target; and

[0015] The matched point cloud data is associated with the category of the first target.

[0016] Optionally, the method further comprises:

[0017] receiving a network signal using a wireless receiver on the vehicle;

[0018] determining the signal quality of received network signals;

[0019] generating a network layer using the signal quality of the network signal; and

[0020] The high-precision map is generated using the network layer.

[0021] Optionally, the network signal includes a cellular signal and / or a Wi-Fi signal.

[0022] Optionally, the plurality of sensors include visual sensors, the method further comprising:

[0023] collecting a plurality of images using the visual sensor;

[0024] generating a basic semantic information layer using the plurality of images; and

[0025] The high-precision layer is generated by utilizing the basic semantic information layer.

[0026] Optionally, determining the driving trajectory of the vehicle comprises determining the driving trajectory of the vehicle using at least one of the following or a combination thereof:

[0027] A combination of inertial navigation and satellite navigation;

[0028] Visual SLAM using images collected by vision sensors; and

[0029] Semantic SLAM using images collected by visual sensors.

[0030] Another aspect of the present disclosure provides an apparatus for generating a multi-layer high-precision map, comprising:

[0031] a module for receiving data from a plurality of sensors on the vehicle, the plurality of sensors including at least a millimeter-wave radar and an ultrasonic radar;

[0032] A module for constructing a driving trajectory of the vehicle based on data collected by the multiple sensors;

[0033] a module for fusing the millimeter-wave point cloud data collected by the millimeter-wave radar and the ultrasonic point cloud data collected by the ultrasonic radar according to the driving trajectory of the vehicle to generate a millimeter-wave-ultrasonic information layer; and

[0034] A module for generating a high-precision map using the millimeter wave-ultrasonic wave information layer.

[0035] Optionally, the module for fusing the millimeter wave point cloud data and the ultrasonic point cloud data includes:

[0036] a module for determining, for each trajectory point on the driving trajectory, a measurement value of the millimeter wave point cloud data and the ultrasonic point cloud data collected at the trajectory point; and

[0037] A module for selecting one of the millimeter wave point cloud data and the ultrasonic point cloud data as the corresponding point cloud data at the trajectory point according to the measurement values ​​of the millimeter wave point cloud data and the ultrasonic point cloud data.

[0038] Optionally, the plurality of sensors include visual sensors, the apparatus further comprising:

[0039] a module for processing images captured by the visual sensor to detect a first target and determine a category of the first target;

[0040] A module for determining point cloud data matching the first target; and

[0041] A module for associating the matched point cloud data with the category of the first target.

[0042] Optionally, the apparatus further comprises:

[0043] a module for receiving network signals using a wireless receiver on the vehicle;

[0044] a module for determining the signal quality of a received network signal;

[0045] means for generating a network layer using the signal quality of the network signal; and

[0046] A module for generating the high-precision map using the network layer.

[0047] Optionally, the network signal includes a cellular signal and / or a Wi-Fi signal.

[0048] Optionally, the plurality of sensors include visual sensors, the apparatus further comprising:

[0049] a module for acquiring a plurality of images using the visual sensor;

[0050] A module for generating a basic semantic information layer using the plurality of images; and

[0051] A module for generating the high-precision layer using the basic semantic information layer.

[0052] Optionally, determining the driving trajectory of the vehicle comprises determining the driving trajectory of the vehicle using at least one of the following or a combination thereof:

[0053] A combination of inertial navigation and satellite navigation;

[0054] Visual SLAM using images collected by vision sensors; and

[0055] Semantic SLAM using images collected by visual sensors.

[0056] One aspect of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory stores program instructions; the processor runs the program instructions to implement the method for generating a multi-layer high-precision map as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A system for generating a multi-layer high-precision map according to various aspects of the present disclosure is shown.

[0058] Figure 2 is a diagram of a sensor module on a vehicle according to aspects of the present disclosure.

[0059] Figure 3 is a diagram of a high-precision map generation apparatus according to various aspects of the present disclosure.

[0060] Figure 4 is a diagram of a trajectory determination unit according to aspects of the present disclosure.

[0061] Figure 5 It is a flowchart for generating a multi-layer high-precision map according to various aspects of the present disclosure.

[0062] Figure 6 is a diagram of an electronic device for target detection according to aspects of the present disclosure. DETAILED DESCRIPTION

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0064] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0065] Figure 1 A system for generating a multi-layer high-precision map according to various aspects of the present disclosure is shown.

[0066] like Figure 1 As shown, the system for generating a multi-layer HD map may include multiple vehicles 102 and a server 104. The multiple vehicles 102 and the server 104 may communicate via a wireless network (e.g., a cellular network, a Wi-Fi network, etc.).

[0067] Each vehicle 102 may be equipped with a plurality of sensors (e.g., visual sensors, millimeter wave radars, ultrasonic radars, network signal units, inertial measurement units, wheel speed meters, GNSS units, etc., as described below). Figure 2 The plurality of sensors can collect various types of data. The vehicle 102 can send the collected data to the server 104 via a wireless network.

[0068] The server 104 may receive data from each vehicle 102 and process it to generate a multi-layered high-precision map, as described below.

[0069] For example, the server 104 may receive data (e.g., basic semantic data, point cloud data, network signal data, etc.) related to each track point on each vehicle 102's driving trajectory. The server 104 may use the data from multiple vehicles on their respective driving trajectories to generate corresponding base layers, point cloud layers, network layers, etc., further forming a multi-layer high-precision map. For example, the track points of multiple vehicles on their respective driving trajectories may be formed into map points on a map.

[0070] During the driving process of the vehicle 102, a multi-layer high-precision map can be received from the server 104 via a wireless network to assist its intelligent driving. For example, when a vehicle 102 enters a parking lot, a multi-layer high-precision map of the current parking lot can be received from the server 104.

[0071] Figure 2 is a diagram of a sensor device 200 on a vehicle according to aspects of the present disclosure.

[0072] like Figure 2 As shown, the sensor device 200 may include a visual sensor 202 , a millimeter wave radar 204 , an ultrasonic radar 206 , a network signal unit 208 , an inertial measurement unit 210 , a wheel speed meter unit 212 , and a GNSS (Global Navigation Satellite System) unit 214 .

[0073] The visual sensor 202 may include multiple cameras and an image processor connected to the cameras. For example, a camera (e.g., a monocular camera, a multi-camera camera, etc.) may be provided at the left front, right front, left rear, and right rear of the vehicle body. The multiple cameras may acquire multiple images in real time. The image processor receives multiple images captured by the multiple cameras at each moment and processes the multiple images. The image processing may include distortion correction, image stitching, etc. For example, the image processor may stitch images captured by multiple cameras to obtain a bird's-eye view of the vehicle. The bird's-eye view may include traffic signs on the road surface, such as arrows, road lines, speed bumps, zebra crossings, parking space lines, etc.

[0074] Millimeter-wave radar 204 detects objects in the millimeter-wave band (30-300 GHz frequency range), emitting millimeter waves into the surrounding environment and receiving millimeter-wave point cloud data reflected back from targets. This millimeter-wave point cloud data can include the distance and orientation of the detected target relative to the vehicle. The detection range of millimeter-wave radar 204 is generally in the range of tens to hundreds of meters.

[0075] Ultrasonic radar 206 utilizes ultrasonic waves for detection, with common operating frequencies including 40kHz, 48kHz, and 58kHz. It transmits ultrasonic waves into the surrounding environment and receives ultrasonic point cloud data reflected back from targets. This ultrasonic point cloud data may include the distance and orientation of the detected target relative to the vehicle. The detection range of ultrasonic radar 206 is generally within ten meters.

[0076] The network signal unit 208 can receive wireless network signals, such as cellular network signals and Wi-Fi signals. Furthermore, the network signal unit 208 can measure the quality of the wireless network signals, such as signal strength, signal delay, etc.

[0077] Inertial measurement unit (IMU) 210 may measure the linear acceleration and angular acceleration of the vehicle.

[0078] The wheel speed meter unit 212 collects data from a wheel speed meter, such as wheel rotation speed.

[0079] The GNSS unit 214 can receive satellite positioning signals. For example, GNSS can include the US GPS, Russia's Glonass, Europe's Galileo, China's BeiDou satellite navigation system, and the like.

[0080] The satellite positioning signals received by the GNSS unit 214 are in a global geocentric coordinate system, which can be converted into a map coordinate system for combination with other positioning information.

[0081] The sensor device 200 may transmit the data collected by the various sensors (as described above) to the server 104 for further processing.

[0082] Figure 3 FIG. 3 is a diagram of a high-precision map generation device 300 according to various aspects of the present disclosure. The high-precision map generation device 300 may be included in the server 104 .

[0083] like Figure 3 As shown, the high-precision map generation device 300 may include a trajectory determination unit 302 , a base layer unit 304 , a point cloud layer unit 306 , and a network layer unit 308 .

[0084] The trajectory determination unit 302 may determine the driving trajectory of the vehicle based on data from various sensors of the vehicle.

[0085] Figure 4 is a diagram of a trajectory determination unit 302 according to aspects of the present disclosure.

[0086] like Figure 4 As shown, the trajectory determination unit 302 may include a combined navigation module 402 , a visual SLAM module 404 , and a semantic SLAM module 406 .

[0087] The integrated navigation module 402 may process data (eg, linear acceleration and angular acceleration data) from the vehicle's inertial measurement unit 210 , wheel speed data from the speedometer unit 212 , and satellite positioning signals from the GNSS unit 214 to generate integrated navigation data.

[0088] Specifically, the integrated navigation module 402 may perform a combination of inertial navigation and satellite navigation.

[0089] Inertial navigation can determine the position of the vehicle (the position in the coordinate system of the map) based on the data (linear acceleration and angular acceleration data) from the inertial measurement unit 210 and the wheel speed data from the speedometer unit 212 .

[0090] Satellite navigation can utilize satellite positioning signals received by the GNSS unit 214 to perform positioning.

[0091] The positioning results of satellite navigation can be converted from the global geocentric coordinate system to the map coordinate system, and then combined with the positioning results of inertial navigation.

[0092] In some indoor scenarios (e.g., underground parking lots), the strength of the satellite positioning signal is unstable, and the vehicle's GNSS unit 214 receives the satellite positioning signal intermittently. In such scenarios, inertial navigation can be used for positioning. During the inertial navigation positioning process, if the GNSS module receives a satellite positioning signal at a certain moment, the satellite signal can be used to correct the positioning result of the inertial navigation. For example, the positioning result in the satellite positioning signal received at that moment can be compared with the positioning result of the inertial navigation. If the distance between the two is greater than a threshold distance, the current positioning result is updated to the positioning result in the satellite positioning signal.

[0093] The visual SLAM (Simultaneous Localization and Mapping) module 404 may use visual SLAM technology to determine the trajectory of the vehicle based on the images obtained by the visual sensor 202 .

[0094] Visual SLAM can include monocular SLAM and binocular SLAM.

[0095] Monocular SLAM can be performed using a single monocular camera. Monocular SLAM can triangulate several images captured by the camera on the vehicle at adjacent times while the vehicle is moving, measure the distance between reference pixels in different images, and thus obtain the vehicle's motion trajectory.

[0096] Binocular SLAM uses parallax between left and right cameras to calculate pixel distances, thereby localizing the vehicle. A binocular camera system consists of two monocular cameras, but the distance between them (called the baseline) is known. This baseline can be used to estimate the spatial position of each pixel, thereby determining the vehicle's trajectory.

[0097] The vehicle positions determined at various times can be combined (e.g., concatenated) to obtain a vehicle trajectory. Each trajectory point on the trajectory is associated with a time, e.g., the time at which the vehicle is determined to be at that trajectory point on the trajectory. In one aspect, the position (position in the coordinate system of the map) of each trajectory point on the trajectory and the corresponding time can be stored as an entry in a memory.

[0098] Furthermore, the visual SLAM module 404 may also associate and store the visual feature map (ie, one or more images captured by the visual sensor 202 ) with each point (and / or corresponding time) on the trajectory while determining the trajectory of the vehicle.

[0099] The semantic SLAM module 406 may use semantic SLAM technology to determine the trajectory of the vehicle based on the images obtained by the vision sensor 202 .

[0100] The semantic SLAM module 406 can obtain one or more images (e.g., a bird's-eye view, as described above) obtained by the visual sensor 202 at each moment, and identify reference targets (e.g., traffic signs) therein based on the one or more images obtained at that moment. For example, a neural network can be used to identify traffic signs (e.g., road signs, ground arrows, road lines, speed bumps, zebra crossings, parking space lines, etc.) in the image as reference targets.

[0101] The travel distance and direction of the vehicle can then be determined based on the position changes of the identified reference targets in a plurality of temporally adjacent images, thereby determining the trajectory of the vehicle.

[0102] Any of the integrated navigation module 402 , the visual SLAM module 404 , and the semantic SLAM module 406 may be used to determine the driving trajectory of the vehicle.

[0103] In one aspect, the satellite positioning signals received by the GNSS unit 214 may be used to adjust the trajectories generated by the visual SLAM module 404 and the semantic SLAM module 406 .

[0104] For example, the GNSS unit 214 may receive satellite positioning signals at time t1 and t2. At time t1, the vehicle's position is determined to be G1 by the satellite positioning signal, and at time t2, the vehicle's position is determined to be G2 by the satellite positioning signal. Further, the distance L between position G1 and position G2 may be determined. G and direction D G .

[0105] Taking the visual SLAM module 404 as an example, at time t1, the vehicle position determined by the visual SLAM module 404 is S1; at time t2, the vehicle position determined by the visual SLAM module 404 is S2. Further, the distance L between position S1 and position S2 can be determined. S and direction D S .

[0106] The distance L can then be determined G and L S The distance difference between them, and the direction D G and D S If the distance difference is greater than the distance threshold and / or the angle difference is greater than the angle threshold, the data of the satellite positioning signal (eg, the distance L G and direction D G ) to adjust (correct) the vehicle's trajectory.

[0107] In one aspect, satellite positioning signal data may be used to provide an initial value of the trajectory for the SLAM module 404. For example, the vehicle position G1 determined by the satellite positioning signal at time t1 is used as the position corresponding to time t1.

[0108] On the other hand, the relative pose changes between satellite positioning signal data can be used as constraints to optimize the factor graph in SLAM, and the poses of key frames in SLAM can be updated through local or global optimization to correct trajectory deviations.

[0109] As an example, the ICP (Iterative Closest Point) algorithm can be used to adjust the vehicle's trajectory using satellite positioning signal data. ICP obtains corresponding point pairs between the source point cloud (vehicle trajectory) and the target point cloud (satellite positioning signal), constructs a rotation and translation matrix based on the corresponding point pairs, and uses the obtained matrix to transform the source point cloud to the coordinate system of the target point cloud. It estimates the error function between the transformed source point cloud and the target point cloud. If the error function value is greater than a threshold, the above operation is iterated until the given error requirement is met. The trajectory determined using the semantic SLAM module 406 can also be adjusted using satellite positioning signals using the above method.

[0110] On the other hand, loop closure adjustment can be performed on the trajectories generated by the visual SLAM module 404 and the semantic SLAM module 406 .

[0111] Taking semantic SLAM as an example, if an object (e.g., an arrow on the ground) is detected in an image at a certain time, and then detected again some time later, it can be determined that the vehicle has returned to its original position, i.e., it has completed a closed loop. This allows the vehicle's trajectory to be adjusted.

[0112] Specifically, a loop closure candidate frame for the current frame can be identified from the historical frames. For example, the distances between multiple historical frames and the current frame can be detected. When the distance between a particular historical frame and the current frame is very small (e.g., less than a threshold), the historical frame can be determined to be a loop closure candidate frame for the current frame. Alternatively, the difference in description information between the historical frames and the current frame can be determined. If the description information of the particular historical frame is similar to that of the current frame, the historical frame can be determined to be a loop closure candidate frame for the current frame.

[0113] In one aspect, multiple loop closure candidate frames may be determined for a current frame and screened in a subsequent process.

[0114] The relative pose relationship between the current frame and the loop closure candidate frame can then be determined, and the determined pose relationship is used as a constraint to adjust the factor graph in SLAM.

[0115] For example, a loop candidate frame and its temporally adjacent key frames can be selected to generate a local map, and the relative pose relationship between the current frame and the loop candidate frame can be used as the initial value to project the semantic information of the current frame into the local map. The overlap rate between the semantic information in the current frame and the semantic information in the local map is calculated, and the pose transformation relationship with the highest overlap rate is found by adjusting near the initial frame. The pose of the current frame is then recalculated as the loop optimization result of the current frame, and the frames between the initial frame and the current frame are adjusted using the SLAM algorithm to achieve loop adjustment.

[0116] Loop closure adjustment for semantic SLAM is particularly suitable for parking lots, where there is a certain probability that the vehicle's driving trajectory will be a closed loop. Through loop closure adjustment, the accumulated errors during the vehicle's closed-loop driving process can be eliminated.

[0117] Back to Figure 3 The base layer unit 304 may associate each point in the vehicle trajectory generated by the trajectory determination unit 302 with its corresponding semantic information. The semantic information may be an object (e.g., a road sign, ground arrow, road marking, speed bump, zebra crossing, parking space marking, etc.) detected from multiple images captured by the visual sensor 202 at that point (e.g., using a neural network).

[0118] If the vehicle's driving trajectory is determined using the semantic SLAM module 406, then a target detected at each trajectory point on the trajectory using semantic SLAM can be directly associated (mapped) with the trajectory point.

[0119] If the vehicle's driving trajectory is determined using the combined navigation module 402 or the visual SLAM module 404, the images collected at each trajectory point (or its corresponding moment) on the driving trajectory can be processed (for example, using a neural network for target recognition) to identify one or more targets therein, and then the one or more targets can be associated with the trajectory point (or its corresponding moment).

[0120] The point cloud layer unit 306 receives the millimeter wave point cloud data from the millimeter wave radar 202 and the ultrasonic point cloud data from the ultrasonic radar 204 , and processes (fuses) the millimeter wave point cloud data and the ultrasonic point cloud data to generate a point cloud layer.

[0121] Millimeter-wave radar 204 has a longer detection range, typically within tens to hundreds of meters. Ultrasonic radar 206 has a shorter detection range, typically within ten meters. Millimeter-wave / ultrasonic point cloud data may include the distance and orientation of detected targets relative to the vehicle, as well as the intensity of reflected waves corresponding to the targets. In one example, the millimeter-wave / ultrasonic point cloud data may be compared with a threshold, and millimeter-wave / ultrasonic point cloud data with reflected wave intensities below the threshold may be filtered out.

[0122] Furthermore, dynamic point cloud data (e.g., point cloud data returned from moving pedestrians or vehicles) can be filtered out, leaving only static point cloud data.

[0123] As an example, the reflected wave intensity in millimeter wave / ultrasonic point cloud data can be used to distinguish objects of different types (eg, materials), thereby filtering out point cloud data related to dynamic objects such as pedestrians and vehicles.

[0124] As another example, velocity information about data points in millimeter-wave point cloud data can be used to filter out dynamic point cloud data based on the velocity information.

[0125] As another example, a dynamic object may be identified through a visual image, the visual image may be associated with millimeter wave / ultrasonic point cloud data, and the point cloud data corresponding to the dynamic object may be filtered out.

[0126] In one aspect, millimeter wave point cloud data and ultrasonic point cloud data can be fused.

[0127] In one example, millimeter-wave point cloud data can be matched with ultrasonic point cloud data to find matching millimeter-wave point cloud data and ultrasonic point cloud data (e.g., corresponding to the same environmental point). This means point cloud data with the same distance and orientation as the vehicle at a certain trajectory point (time). The reflected wave intensities of the matching millimeter-wave point cloud data and the ultrasonic point cloud data can then be compared. The one with the stronger reflected wave intensity is selected as the point cloud data corresponding to the vehicle at that trajectory point and the environmental point.

[0128] At certain environmental points, only one of ultrasonic point cloud data and millimeter wave point cloud data exists, and this point cloud data can be used as the point cloud data of the environmental point.

[0129] The point cloud data generated for each track point on the driving track can then be associated with the track point to generate a point cloud layer. The point cloud data can include distance to the vehicle, orientation, reflection intensity, etc.

[0130] By fusing millimeter-wave and ultrasonic point cloud data, point cloud data can be expanded to cover ranges from a few meters to tens to hundreds of meters. This fusion reduces the amount of point cloud data and improves its accuracy. Furthermore, while millimeter-wave detection can produce some jumps, ultrasonic detection produces very stable contour points at close range, enhancing the stability of the point cloud layer.

[0131] In one aspect, the point cloud data in the point cloud layer may be classified using image data collected by a visual sensor.

[0132] For example, for a specific track point on the driving track, the image data collected by the visual sensor can be processed to identify one or more targets therein (for example, using a neural network for target recognition, as described above).

[0133] For each identified target, a determination is made as to whether there is matching target point cloud data (e.g., the identified target and the point cloud data are at the same distance from the vehicle, at the same orientation relative to the vehicle's central axis, etc.) in the point cloud data for that track point (i.e., the point cloud data collected at that track point, such as millimeter wave data or ultrasonic data). If there is point cloud data that matches the target in the image, the determined category (the category of the target in the image, as determined above) can be associated with the corresponding point cloud data.

[0134] The point cloud data in the point cloud layer is classified through image data. In the resulting high-precision multi-layer map, each map point can include the point cloud data at that map point (for example, a target detected by radar) and its type (for example, a road sign, a pillar, etc.).

[0135] The network layer unit 308 may receive quality data of wireless signals (e.g., signals of a cellular network or a Wi-Fi network) from the network signal unit 208. Furthermore, the network signal quality data of each point on the trajectory may be associated with the point, thereby forming a network layer.

[0136] Through the network layer, vehicles can avoid areas with poor network signal quality during path planning, thereby obtaining a better autonomous driving experience.

[0137] This application first determines the driving trajectory of each vehicle for each vehicle, and then uses each trajectory point on the driving trajectory and its corresponding time as a benchmark to associate the data sensed by each sensor (visual sensor, millimeter wave / ultrasonic radar, network signal unit, etc.) with the trajectory point / time. The trajectories of multiple vehicles and their corresponding data are then combined (for example, the trajectory points of multiple vehicles and their corresponding data are projected onto the coordinate system of the map), thereby forming a multi-layer high-precision map. At each map point in the multi-layer high-precision map, there are corresponding multiple types of data (for example, image data, point cloud data, network signal data, etc.).

[0138] Figure 5 It is a flowchart for generating a multi-layer high-precision map according to various aspects of the present disclosure.

[0139] like Figure 5 As shown, in step 502 , data from a plurality of sensors on a vehicle may be received, the plurality of sensors including at least a millimeter wave radar and an ultrasonic radar.

[0140] In step 504 , the driving trajectory of the vehicle may be constructed based on the data collected by the multiple sensors.

[0141] In one aspect, determining the vehicle's driving trajectory includes determining the vehicle's driving trajectory using at least one of the following or a combination thereof: a combination of inertial navigation and satellite navigation; visual SLAM performed using images collected by a visual sensor; and semantic SLAM performed using images collected by a visual sensor.

[0142] In step 506 , the millimeter wave point cloud data collected by the millimeter wave radar and the ultrasonic point cloud data collected by the ultrasonic radar may be fused according to the driving trajectory of the vehicle to generate a millimeter wave-ultrasonic wave information layer.

[0143] On the one hand, fusing the millimeter wave point cloud data and the ultrasonic point cloud data may include: determining, for each trajectory point on the driving trajectory, measurement values ​​of the millimeter wave point cloud data and the ultrasonic point cloud data collected at the trajectory point; and selecting one of the millimeter wave point cloud data and the ultrasonic point cloud data as the corresponding point cloud data at the trajectory point based on the measurement values ​​of the millimeter wave point cloud data and the ultrasonic point cloud data.

[0144] The measurement value of millimeter wave / ultrasonic point cloud data can be the reflected wave intensity, signal-to-noise ratio, distance and angle range of the point cloud data, etc. For example, the millimeter wave / ultrasonic point cloud data can be the weighted sum of the reflected wave intensity, signal-to-noise ratio, distance and angle range.

[0145] In one aspect, the multiple sensors include a visual sensor, and the method further includes: processing images captured by the visual sensor to detect a first target and determine a category of the first target; determining point cloud data matching the first target; and associating the matched point cloud data with the category of the first target.

[0146] In step 508, the millimeter wave-ultrasonic wave information layer may be used to generate a high-precision map.

[0147] In one aspect, the method further comprises: receiving a network signal using a wireless receiver on the vehicle;

[0148] Determine the signal quality of the received network signal; use the signal quality of the network signal to generate a network layer; and use the network layer to generate the high-precision map.

[0149] In one aspect, the network signal comprises a cellular signal and / or a wifi signal.

[0150] In one aspect, the multiple sensors include a visual sensor, and the method further includes: using the visual sensor to collect multiple images; using the multiple images to generate a basic semantic information layer; and using the basic semantic information layer to generate the high-precision layer.

[0151] Figure 6 is a diagram of an electronic device for target detection according to aspects of the present disclosure.

[0152] like Figure 6 As shown, the electronic device 600 may include a memory 602 and a processor 604. The memory 602 stores program instructions, and the processor 604 can be connected to and communicate with the memory 602 via a bus 606. The processor 604 can call the program instructions in the memory 602 to perform the following steps: receiving data from multiple sensors on the vehicle, the multiple sensors including at least a millimeter-wave radar and an ultrasonic radar; constructing a driving trajectory of the vehicle based on the data collected by the multiple sensors; based on the driving trajectory of the vehicle, fusing the millimeter-wave point cloud data collected by the millimeter-wave radar and the ultrasonic point cloud data collected by the ultrasonic radar to generate a millimeter-wave-ultrasonic information layer; and generating a high-precision map using the millimeter-wave-ultrasonic information layer.

[0153] Optionally, the processor 604 can also call the program instructions in the memory 602 to perform the following steps: for each trajectory point on the driving trajectory, determine the measurement values ​​of the millimeter wave point cloud data and the ultrasonic point cloud data collected at the trajectory point; and select one of the millimeter wave point cloud data and the ultrasonic point cloud data as the corresponding point cloud data at the trajectory point based on the measurement values ​​of the millimeter wave point cloud data and the ultrasonic point cloud data.

[0154] Optionally, the processor 604 can also call program instructions in the memory 602 to perform the following steps: processing the image captured by the visual sensor to detect the first target and determine the category of the first target; determining point cloud data matching the first target; and associating the matched point cloud data with the category of the first target.

[0155] Optionally, the processor 604 can also call program instructions in the memory 602 to perform the following steps: using the wireless receiver on the vehicle to receive a network signal; determining the signal quality of the received network signal; using the signal quality of the network signal to generate a network layer; and using the network layer to generate the high-precision map.

[0156] Optionally, the network signal includes a cellular signal and / or a wifi signal.

[0157] Optionally, the processor 604 may also call program instructions in the memory 602 to perform the following steps: using a visual sensor to capture multiple images; using the multiple images to generate a basic semantic information layer; and using the basic semantic information layer to generate the high-precision layer.

[0158] Optionally, the processor 604 may also call program instructions in the memory 602 to perform the following steps: a combination of inertial navigation and satellite navigation; visual SLAM using images collected by the visual sensor; and semantic SLAM using images collected by the visual sensor.

[0159] The description set forth herein in conjunction with the accompanying drawings describes example configurations and does not represent all examples that can be implemented or fall within the scope of the claims. The term "exemplary" as used herein means "serving as an example, instance, or illustration" and does not mean "better than" or "better than other examples." This detailed description includes specific details to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described examples.

[0160] In the accompanying drawings, similar components or features may have the same reference number. In addition, components of the same type may be distinguished by following the reference number with a dash and a second reference number that distinguishes between the similar components. If only the first reference number is used in the specification, the description applies to any of the similar components having the same first reference number, regardless of the second reference number.

[0161] The various illustrative blocks and modules described in conjunction with the disclosure herein may be implemented or executed with a general purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration).

[0162] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Other examples and implementations fall within the scope of this disclosure and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwiring, or any combination thereof. Features implementing the functions may also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. In addition, as used herein (including in the claims), "or" used in an enumeration of items (e.g., an enumeration of items with phrases such as "at least one of" or "one or more of") indicates an inclusive enumeration, such that, for example, an enumeration of at least one of A, B, or C means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). Similarly, as used herein, the phrase "based on" should not be interpreted as referring to a closed set of conditions. For example, an exemplary step described as "based on condition A" may be based on both condition A and condition B without departing from the scope of the present disclosure. In other words, as used herein, the phrase "based on" should be interpreted in the same manner as the phrase "based at least in part on."

[0163] Computer-readable media include both non-transient computer storage media and communication media, which include any media that facilitates a computer program to be transferred from one place to another. Non-transient storage media can be any available medium that can be accessed by a general or special-purpose computer. As an example and not limitation, non-transient computer-readable media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), compact disc (CD) ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other non-transient medium that can be used to carry or store the desired program code means of an instruction or data structure form and can be accessed by a general or special-purpose computer or a general or special-purpose processor. Any connection is also properly referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. Disk and disc, as used herein, include CDs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of computer-readable media.

[0164] The description herein is provided to enable those skilled in the art to make or use the present disclosure. Various modifications to the present disclosure will be apparent to those skilled in the art, and the universal principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a multi-layer high-precision map, comprising: receiving data from a plurality of sensors on the vehicle, the plurality of sensors including at least a millimeter-wave radar and an ultrasonic radar; constructing a driving trajectory of the vehicle based on data collected by the multiple sensors; According to the driving trajectory of the vehicle, the millimeter wave point cloud data collected by the millimeter wave radar and the ultrasonic point cloud data collected by the ultrasonic radar are fused to generate a millimeter wave-ultrasonic wave information layer; and The millimeter wave-ultrasonic wave information layer is used to generate a high-precision map. The fusing of the millimeter wave point cloud data and the ultrasonic point cloud data comprises: For each trajectory point on the driving trajectory, determining a measurement value of the millimeter wave point cloud data and the ultrasonic point cloud data collected at the trajectory point; and selecting one of the millimeter wave point cloud data and the ultrasonic point cloud data as the corresponding point cloud data at the trajectory point according to the measurement values ​​of the millimeter wave point cloud data and the ultrasonic point cloud data, and Wherein the plurality of sensors include a visual sensor, the method further comprises: processing the image captured by the visual sensor to detect a first object and determine a category of the first object; determining point cloud data matching the first target; and The matched point cloud data is associated with the category of the first target.

2. The method of claim 1, further comprising: receiving a network signal using a wireless receiver on the vehicle; determining the signal quality of received network signals; generating a network layer using the signal quality of the network signal; as well as The high-precision map is generated using the network layer.

3. The method of claim 2, wherein the network signal comprises a cellular signal and / or a Wi-Fi signal.

4. The method of claim 1 , wherein the plurality of sensors include visual sensors, the method further comprising: collecting a plurality of images using the visual sensor; generating a basic semantic information layer using the plurality of images; as well as The high-precision map is generated using the basic semantic information layer.

5. The method of claim 1 , wherein determining the driving trajectory of the vehicle comprises determining the driving trajectory of the vehicle using at least one of the following or a combination thereof: A combination of inertial navigation and satellite navigation; Visual SLAM using images collected by vision sensors; and Semantic SLAM using images collected by visual sensors.

6. A device for generating a multi-layer high-precision map, comprising: a module for receiving data from a plurality of sensors on the vehicle, the plurality of sensors including at least a millimeter-wave radar and an ultrasonic radar; A module for constructing a driving trajectory of the vehicle based on data collected by the multiple sensors; a module for fusing the millimeter-wave point cloud data collected by the millimeter-wave radar and the ultrasonic point cloud data collected by the ultrasonic radar according to the driving trajectory of the vehicle to generate a millimeter-wave-ultrasonic information layer; and A module for generating a high-precision map using the millimeter wave-ultrasonic wave information layer, The module for fusing the millimeter wave point cloud data and the ultrasonic point cloud data includes: a module for determining, for each trajectory point on the driving trajectory, a measurement value of the millimeter wave point cloud data and the ultrasonic point cloud data collected at the trajectory point; and a module for selecting one of the millimeter wave point cloud data and the ultrasonic point cloud data as the corresponding point cloud data at the trajectory point according to the measurement values ​​of the millimeter wave point cloud data and the ultrasonic point cloud data, and Wherein the plurality of sensors include a visual sensor, the apparatus further comprises: a module for processing images captured by the visual sensor to detect a first target and determine a category of the first target; A module for determining point cloud data matching the first target; and A module for associating the matched point cloud data with the category of the first target.

7. The apparatus of claim 6, further comprising: a module for receiving network signals using a wireless receiver on the vehicle; a module for determining the signal quality of a received network signal; a module for generating a network layer using the signal quality of the network signal; as well as A module for generating the high-precision map using the network layer. The apparatus of claim 7 , wherein the network signal comprises a cellular signal and / or a Wi-Fi signal.

9. The apparatus of claim 6, wherein the plurality of sensors include visual sensors, the apparatus further comprising: a module for acquiring a plurality of images using the visual sensor; A module for generating a basic semantic information layer using the plurality of images; as well as A module for generating the high-precision map using the basic semantic information layer.

10. The apparatus of claim 6, wherein determining the driving trajectory of the vehicle comprises determining the driving trajectory of the vehicle using at least one of the following or a combination thereof: A combination of inertial navigation and satellite navigation; Visual SLAM using images collected by vision sensors; and Semantic SLAM using images collected by visual sensors.

11. An electronic device comprising a processor and a memory, wherein the memory stores program instructions; the processor executes the program instructions to implement the method for generating a multi-layer high-precision map as described in any one of claims 1 to 5.

Images