Local semantic map construction method and system based on high-speed scene and medium

Abstract

The invention relates to a local semantic map construction method and system based on a high-speed scene and a medium, and the method comprises the steps: U1, obtaining the image data information of a road in real time based on a vehicle-mounted camera, obtaining the data information of the point cloud of the road in real time based on a vehicle-mounted millimeter wave radar, and obtaining the data information of the point cloud of the road; acquiring data information of angular velocity and acceleration of the vehicle in real time based on a vehicle-mounted IMU, and acquiring data information of speed of the vehicle in real time based on a vehicle-mounted wheel speed sensor; and U2, based on the data information of the point cloud of the road and the image data information of the road, performing semantic fusion on the image and the point cloud of the road by adopting an improved visual BEV semantic segmentation model to obtain semantic data information of the road. According to the method, dependence on a high-precision map can be avoided, vehicle planning control is carried out by adopting the vehicle end sensing result, the cost is greatly reduced, the vehicle end sensing stability is improved, and a good effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of semantic map construction, and in particular to a method, system and medium for constructing a local semantic map based on a high-speed scenario. Background Art

[0002] In recent years, the demand for autonomous driving has become increasingly urgent. To achieve autonomous capabilities, vehicles are equipped with a variety of sensors, such as GPS, IMUs, cameras, LiDAR, millimeter-wave radar, and wheel odometers. Positioning is a fundamental function of autonomous driving systems. High-precision positioning relies on high-precision sensors and high-definition maps (HD maps). Currently, RTK-GPS and LiDAR are two commonly used sensors, widely used for centimeter-level positioning. RTK-GPS receives signals from satellites and ground-based base stations to provide accurate global pose in open areas. LiDAR collects point clouds of the surrounding environment. Through point cloud matching, vehicles can locate themselves within HD maps even in environments without GPS signals. These methods have been used in robotaxi applications in many cities. Solutions based on LiDAR and HD maps are ideal for robotaxi applications. However, several drawbacks limit their use in general-purpose, mass-produced vehicles. First, mass-produced vehicles cannot afford the high cost of LiDAR and HD maps. Furthermore, point cloud maps consume a large amount of memory, making them prohibitive for mass production. HD maps are labor-intensive to produce and difficult to maintain timely updates. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method, system and medium for constructing a local semantic map based on high-speed scenarios. It not only breaks away from the dependence on high-precision maps, but also uses vehicle-side perception results for vehicle planning and control, greatly reducing costs and improving vehicle-side perception stability, achieving good results.

[0004] In order to achieve the above-mentioned and other related purposes, the present invention provides the following technical solutions:

[0005] A method for constructing a local semantic map based on a high-speed scene, the method comprising:

[0006] U1. A vehicle traveling on a highway acquires real-time road image data using an onboard camera, road point cloud data using an onboard millimeter-wave radar, angular velocity and acceleration data using an onboard IMU, and vehicle speed data using an onboard wheel speed sensor.

[0007] U2. Based on the data information of the road point cloud and the image data information of the road, an improved visual BEV semantic segmentation model is used to semantically fuse the road image and point cloud to obtain semantic data information of the road;

[0008] U3. Based on the semantic data information of the road, the single-frame semantic information is verified, abnormal outliers are removed, and the result is converted into single-frame semantic point cloud information, and semantic tracking is performed to obtain the semantic point cloud data information of the road features;

[0009] U4. Based on the data information of the vehicle's angular velocity, acceleration and speed, a multi-model feature fusion algorithm is used to characterize the relative position of the vehicle to obtain data information of the relative position of the vehicle;

[0010] U5. Based on the data information of the relative position of the vehicle and the data information of the characteristic stable semantic point cloud of the road, an improved factor graph fusion algorithm is used to characterize the local semantic map to obtain data information of the local semantic map of the road.

[0011] Furthermore, in step U2, the semantic fusion of the road image and point cloud using the improved visual BEV semantic segmentation model includes:

[0012] U21. Based on the image data of the road, a Resnet-101 network is used to extract features from each image. A 3D space is then defined, and each subgrid is projected onto the feature map. Features are bilinearly sampled to obtain a 3D feature grid for the road image.

[0013] U22 converts the road point cloud data information into an image, the millimeter wave image feature dimension is 15 dimensions, and obtains the road point cloud image feature data information;

[0014] U23. Based on the data information of the image features of the 3D feature grid of the road image and the image features of the point cloud of the road, a fusion function Q of the road image and the point cloud is established.

[0015] ,

[0016] Among them, x is the data information of the 3D feature grid of the road image, y is the data information of the image features of the road point cloud, ɑ1, ɑ2, and ɑ3 are the weight factors for the feature fusion of the road image and point cloud. The road image and point cloud are semantically fused and compressed into a BEV feature map. The BEV feature map is then processed using the Resnet-18 network to obtain the semantic data information of the road.

[0017] Furthermore, the weight factors ɑ1, ɑ2 and ɑ3 of the feature fusion of the road image and point cloud are,

[0018] ,

[0019] ,

[0020] ,

[0021] Among them, x is the data information of the 3D feature grid of the road image, and y is the data information of the image features of the point cloud of the road.

[0022] Furthermore, in step U4, the characterization of the relative position of the vehicle using the multi-model feature fusion algorithm includes:

[0023] U41. The vehicle's angular velocity and acceleration data are input into the IMU pre-integration model to characterize the vehicle's first posture to obtain the vehicle's first posture data;

[0024] U42. The angular velocity and speed data of the vehicle are input into the wheel speed odometer model to characterize the second position of the vehicle to obtain the data information of the second position of the vehicle;

[0025] U43. Based on the data information of the first posture of the vehicle and the data information of the second posture of the vehicle, establish the relative posture function W of the vehicle,

[0026] ,

[0027] Among them, z1 is the data information of the first posture of the vehicle, z2 is the data information of the second posture of the vehicle, β1, β2 and β3 are any constant parameters between 0 and 1, which characterize the relative posture of the vehicle and obtain the data information of the relative posture of the vehicle.

[0028] Furthermore, the constraint function f of the constant parameter is,

[0029] ,

[0030] The value range of the constraint function f is (0,1).

[0031] Furthermore, in step U5, the characterization of the local semantic map using the improved factor graph fusion algorithm includes:

[0032] U51. Based on the data information of the relative position of the vehicle, establish a local factor function G of the relative position of the vehicle i ,

[0033] ,

[0034] Among them, r iFor the relative position of the i-th position point of the vehicle, the local factor nodes of the factor graph are represented to obtain the data information of the local factor nodes of the factor graph;

[0035] U52. Based on the data information of the semantic point cloud with stable characteristics of the road, establish the local variable node function H of the road i ,

[0036] ,

[0037] Among them, q i For the i-th semantic point cloud with stable road features, the local variable nodes of the factor graph are represented to obtain the data information of the local variable nodes of the factor graph;

[0038] U53. Based on the data information of the local variable nodes of the factor graph and the data information of the local factor nodes of the factor graph, establish a factor graph fusion function S,

[0039] ,

[0040] Among them, g j is the data information of the local variable node of the factor graph, h j is the data information of the local factor node of the factor graph, n is the sample capacity, η j As the fusion factor, the local semantic map is represented to obtain the data information of the local semantic map of the road.

[0041] Furthermore, the fusion factor η j The constraints are,

[0042] .

[0043] Furthermore, the fusion factor η j for,

[0044] ,

[0045] Among them, g j is the data information of the local variable node of the factor graph, h j It is the data information of the local factor nodes of the factor graph.

[0046] In order to achieve the above-mentioned objectives and other related objectives, the present invention also provides a local semantic map construction system based on high-speed scenes, including a computer device that is programmed or configured to execute any one of the steps of the local semantic map construction method based on high-speed scenes.

[0047] In order to achieve the above-mentioned objectives and other related objectives, the present invention also provides a computer-readable storage medium, which stores a computer program programmed or configured to execute any one of the methods for constructing a local semantic map based on a high-speed scene.

[0048] The present invention has the following positive effects:

[0049] 1. The present invention adopts an improved visual BEV semantic segmentation model to perform semantic fusion on road images and point clouds, and combines it with verification of single-frame semantic information, removes abnormal outliers, and converts the results into single-frame semantic point cloud information, and performs semantic tracking. It not only retains historical frames through multi-frame splicing methods and removes single-frame perception error results to obtain a sliding window local semantic map, providing environmental information for planning and control, and breaking away from high-precision maps, but also realizes map-free autonomous driving and reduces dependence on high-precision maps.

[0050] 2. The present invention adopts a multi-model feature fusion algorithm to characterize the relative position of the vehicle, and combines it with an improved factor graph fusion algorithm to characterize the local semantic map. It not only breaks away from the dependence on high-precision maps, but also uses the vehicle-side perception results for vehicle planning and control, greatly reducing costs and improving vehicle-side perception stability, achieving good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart for constructing a local semantic map of the present invention;

[0052] Figure 2 is a schematic diagram of a local semantic map of the present invention;

[0053] Figure 3 Schematic diagram of the method flow of the present invention;

[0054] Figure 4 Schematic diagram of the process of the improved visual BEV semantic segmentation model of the present invention;

[0055] Figure 5 Schematic diagram of the process of the multi-model feature fusion algorithm of the present invention;

[0056] Figure 6 Schematic diagram of the process of the improved factor graph fusion algorithm of the present invention. DETAILED DESCRIPTION

[0057] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0058] Example 1: Figure 1 or Figure 2 or Figure 3 As shown, a method for constructing a local semantic map based on a high-speed scene includes:

[0059] U1. A vehicle traveling on a highway acquires real-time road image data using an onboard camera, road point cloud data using an onboard millimeter-wave radar, angular velocity and acceleration data using an onboard IMU, and vehicle speed data using an onboard wheel speed sensor.

[0060] U2. Based on the data information of the road point cloud and the image data information of the road, an improved visual BEV semantic segmentation model is used to semantically fuse the road image and point cloud to obtain semantic data information of the road;

[0061] U3. Based on the semantic data information of the road, the single-frame semantic information is verified, abnormal outliers are removed, and the result is converted into single-frame semantic point cloud information, and semantic tracking is performed to obtain the semantic point cloud data information of the road features;

[0062] U4. Based on the data information of the vehicle's angular velocity, acceleration and speed, a multi-model feature fusion algorithm is used to characterize the relative position of the vehicle to obtain data information of the relative position of the vehicle;

[0063] U5. Based on the data information of the relative position of the vehicle and the data information of the characteristic stable semantic point cloud of the road, an improved factor graph fusion algorithm is used to characterize the local semantic map to obtain data information of the local semantic map of the road.

[0064] In this embodiment, if Figure 4 As shown, in step U2, the semantic fusion of the road image and point cloud using the improved visual BEV semantic segmentation model includes:

[0065] U21. Based on the image data of the road, a Resnet-101 network is used to extract features from each image. A 3D space is then defined, and each subgrid is projected onto the feature map. Features are bilinearly sampled to obtain a 3D feature grid for the road image.

[0066] U22 converts the road point cloud data information into an image, the millimeter wave image feature dimension is 15 dimensions, and obtains the road point cloud image feature data information;

[0067] U23. Based on the data information of the image features of the 3D feature grid of the road image and the image features of the point cloud of the road, a fusion function Q of the road image and the point cloud is established.

[0068] ,

[0069] Among them, x is the data information of the 3D feature grid of the road image, y is the data information of the image features of the road point cloud, ɑ1, ɑ2, and ɑ3 are the weight factors for the feature fusion of the road image and point cloud. The road image and point cloud are semantically fused and compressed into a BEV feature map. The BEV feature map is then processed using the Resnet-18 network to obtain the semantic data information of the road.

[0070] In this embodiment, the weight factors ɑ1, ɑ2 and ɑ3 of the feature fusion of the road image and the point cloud are:

[0071] ,

[0072] ,

[0073] ,

[0074] Among them, x is the data information of the 3D feature grid of the road image, and y is the data information of the image features of the point cloud of the road.

[0075] This embodiment addresses the problem of traditional ORB descriptor methods failing to accurately reflect road characteristics, making them unsuitable for high-speed scenarios and only suitable for low-speed scenarios. Using a bird's-eye view (BEV) method, through multi-camera fusion and a Transformer model, it can perceive the surrounding environment 20 meters or more in real time, making it suitable for autonomous driving scenarios with frequently changing road conditions. However, BEV can only recognize semantic information in a single frame, which can lead to false detections and missed detections, and it cannot retain historical frame results, resulting in poor stability.

[0076] Example 2: Based on the method for constructing a local semantic map based on a high-speed scene in Example 1, the present invention is further illustrated and described below.

[0077] like Figure 1 or Figure 2 or Figure 3 As shown, a method for constructing a local semantic map based on a high-speed scene includes:

[0078] U1. A vehicle traveling on a highway acquires real-time road image data using an onboard camera, road point cloud data using an onboard millimeter-wave radar, angular velocity and acceleration data using an onboard IMU, and vehicle speed data using an onboard wheel speed sensor.

[0079] U2. Based on the data information of the road point cloud and the image data information of the road, an improved visual BEV semantic segmentation model is used to semantically fuse the road image and point cloud to obtain semantic data information of the road;

[0080] U3. Based on the semantic data information of the road, the single-frame semantic information is verified, abnormal outliers are removed, and the result is converted into single-frame semantic point cloud information, and semantic tracking is performed to obtain the semantic point cloud data information of the road features;

[0081] U4. Based on the data information of the vehicle's angular velocity, acceleration and speed, a multi-model feature fusion algorithm is used to characterize the relative position of the vehicle to obtain data information of the relative position of the vehicle;

[0082] U5. Based on the data information of the relative position of the vehicle and the data information of the characteristic stable semantic point cloud of the road, an improved factor graph fusion algorithm is used to characterize the local semantic map to obtain data information of the local semantic map of the road.

[0083] In this embodiment, if Figure 5 As shown, in step U4, the use of a multi-model feature fusion algorithm to characterize the relative position of the vehicle includes:

[0084] U41. The vehicle's angular velocity and acceleration data are input into the IMU pre-integration model to characterize the vehicle's first posture to obtain the vehicle's first posture data;

[0085] U42. The angular velocity and speed data of the vehicle are input into the wheel speed odometer model to characterize the second position of the vehicle to obtain the data information of the second position of the vehicle;

[0086] U43. Based on the data information of the first posture of the vehicle and the data information of the second posture of the vehicle, establish the relative posture function W of the vehicle,

[0087] ,

[0088] Among them, z1 is the data information of the first posture of the vehicle, z2 is the data information of the second posture of the vehicle, β1, β2 and β3 are any constant parameters between 0 and 1, which characterize the relative posture of the vehicle and obtain the data information of the relative posture of the vehicle.

[0089] In this embodiment, the constraint function f of the constant parameter is,

[0090] ,

[0091] The value range of the constraint function f is (0,1).

[0092] In this embodiment, if Figure 6 As shown, in step U5, the use of the improved factor graph fusion algorithm to characterize the local semantic map includes:

[0093] U51. Based on the data information of the relative position of the vehicle, establish a local factor function G of the relative position of the vehicle i ,

[0094] ,

[0095] Among them, r i For the relative position of the i-th position point of the vehicle, the local factor nodes of the factor graph are represented to obtain the data information of the local factor nodes of the factor graph;

[0096] U52. Based on the data information of the semantic point cloud with stable characteristics of the road, establish the local variable node function H of the road i ,

[0097] ,

[0098] Among them, q i For the i-th semantic point cloud with stable road features, the local variable nodes of the factor graph are represented to obtain the data information of the local variable nodes of the factor graph;

[0099] U53. Based on the data information of the local variable nodes of the factor graph and the data information of the local factor nodes of the factor graph, establish a factor graph fusion function S,

[0100] ,

[0101] Among them, g j is the data information of the local variable node of the factor graph, h j is the data information of the local factor node of the factor graph, n is the sample capacity, η j As the fusion factor, the local semantic map is represented to obtain the data information of the local semantic map of the road.

[0102] In this embodiment, the fusion factor η j The constraints are,

[0103] .

[0104] In this embodiment, the fusion factor η j for,

[0105] ,

[0106] Among them, g j is the data information of the local variable node of the factor graph, h j It is the data information of the local factor nodes of the factor graph.

[0107] In this embodiment, the present invention provides a system for constructing a local semantic map based on a high-speed scene, including a computer device programmed or configured to execute any one of the steps of the method for constructing a local semantic map based on a high-speed scene.

[0108] In this embodiment, the present invention provides a computer-readable storage medium storing a computer program programmed or configured to execute any one of the methods for constructing a local semantic map based on a high-speed scene.

[0109] Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0110] In summary, the present invention not only eliminates the reliance on high-precision maps, but also uses vehicle-side perception results for vehicle planning and control, which greatly reduces costs, improves vehicle-side perception stability, and achieves good results.

[0111] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for constructing a local semantic map based on a high-speed scene, characterized in that: The method comprises: U1. A vehicle traveling on a highway acquires real-time road image data using an onboard camera, road point cloud data using an onboard millimeter-wave radar, angular velocity and acceleration data using an onboard IMU, and vehicle speed data using an onboard wheel speed sensor. U2. Based on the data information of the road point cloud and the image data information of the road, an improved visual BEV semantic segmentation model is used to semantically fuse the road image and point cloud to obtain semantic data information of the road; U3. Based on the semantic data information of the road, the single-frame semantic information is verified, abnormal outliers are removed, and the result is converted into single-frame semantic point cloud information, and semantic tracking is performed to obtain the semantic point cloud data information of the road features; U4. Based on the data information of the vehicle's angular velocity, acceleration and speed, a multi-model feature fusion algorithm is used to characterize the relative position of the vehicle to obtain data information of the relative position of the vehicle; U5. Based on the data information of the relative position of the vehicle and the data information of the characteristic stable semantic point cloud of the road, an improved factor graph fusion algorithm is used to characterize the local semantic map to obtain data information of the local semantic map of the road.

2. The method for constructing a local semantic map based on a high-speed scene according to claim 1, characterized in that: In step U2, the semantic fusion of the road image and point cloud using the improved visual BEV semantic segmentation model includes: U21. Based on the image data of the road, a Resnet-101 network is used to extract features from each image. A 3D space is then defined, and each subgrid is projected onto the feature map. Features are bilinearly sampled to obtain a 3D feature grid for the road image. U22 converts the road point cloud data information into an image, the millimeter wave image feature dimension is 15 dimensions, and obtains the road point cloud image feature data information; U23. Based on the data information of the image features of the 3D feature grid of the road image and the image features of the point cloud of the road, a fusion function Q of the road image and the point cloud is established. ， Among them, x is the data information of the 3D feature grid of the road image, y is the data information of the image features of the road point cloud, ɑ1, ɑ2, and ɑ3 are the weight factors for the feature fusion of the road image and point cloud. The road image and point cloud are semantically fused and compressed into a BEV feature map. The BEV feature map is then processed using the Resnet-18 network to obtain the semantic data information of the road.

3. The method for constructing a local semantic map based on a high-speed scene according to claim 2, characterized in that: The weight factors ɑ1, ɑ2 and ɑ3 of the feature fusion of the road image and point cloud are: ， ， ， Among them, x is the data information of the 3D feature grid of the road image, and y is the data information of the image features of the point cloud of the road.

4. The method for constructing a local semantic map based on a high-speed scene according to claim 1, characterized in that: In step U4, the characterization of the relative position of the vehicle using the multi-model feature fusion algorithm includes: U41. The vehicle's angular velocity and acceleration data are input into the IMU pre-integration model to characterize the vehicle's first posture to obtain the vehicle's first posture data; U42. The angular velocity and speed data of the vehicle are input into the wheel speed odometer model to characterize the second position of the vehicle to obtain the data information of the second position of the vehicle; U43. Based on the data information of the first posture of the vehicle and the data information of the second posture of the vehicle, establish the relative posture function W of the vehicle, ， Among them, z1 is the data information of the first posture of the vehicle, z2 is the data information of the second posture of the vehicle, β1, β2 and β3 are any constant parameters between 0 and 1, which characterize the relative posture of the vehicle and obtain the data information of the relative posture of the vehicle.

5. The method for constructing a local semantic map based on a high-speed scene according to claim 4, characterized in that: The constraint function f of the constant parameter is, ， The value range of the constraint function f is (0,1).

6. The method for constructing a local semantic map based on a high-speed scene according to claim 1, characterized in that: In step U5, the characterization of the local semantic map using the improved factor graph fusion algorithm includes: U51. Based on the data information of the relative position of the vehicle, establish a local factor function G of the relative position of the vehicle i , ， Among them, r i For the relative position of the i-th position point of the vehicle, the local factor nodes of the factor graph are represented to obtain the data information of the local factor nodes of the factor graph; U52. Based on the data information of the semantic point cloud with stable characteristics of the road, establish the local variable node function H of the road i , ， Among them, q i For the i-th semantic point cloud with stable road features, the local variable nodes of the factor graph are represented to obtain the data information of the local variable nodes of the factor graph; U53. Based on the data information of the local variable nodes of the factor graph and the data information of the local factor nodes of the factor graph, establish a factor graph fusion function S, ， Among them, g j is the data information of the local variable node of the factor graph, h j is the data information of the local factor node of the factor graph, n is the sample capacity, η j As the fusion factor, the local semantic map is represented to obtain the data information of the local semantic map of the road.

7. The method for constructing a local semantic map based on a high-speed scene according to claim 6, characterized in that: The fusion factor η j The constraints are, 。 8. The method for constructing a local semantic map based on a high-speed scene according to claim 6, characterized in that: The fusion factor η j for, ， Among them, g j is the data information of the local variable node of the factor graph, h j It is the data information of the local factor nodes of the factor graph.

9. A local semantic map construction system based on high-speed scenes, comprising a computer device, characterized in that: The computer device is programmed or configured to execute the steps of the method for constructing a local semantic map based on a high-speed scene as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program programmed or configured to execute the method for constructing a local semantic map based on a high-speed scene according to any one of claims 1 to 8.

Images