Semantic positioning method combining dynamic and static features, electronic device and medium

Abstract

This invention provides a semantic localization method, electronic device, and machine-readable storage medium that combine dynamic and static features. The semantic localization method includes: real-time acquisition of 3D sensing information of the road where the vehicle is located by sensors; processing the 3D sensing information to obtain road semantic information, which includes static and dynamic semantic features; generating 3D road observation data including static and dynamic semantic features based on the road semantic information; semantically registering the road 3D observation data with a pre-constructed feature map; and obtaining the optimal registration pose through a cost function optimization method, thereby achieving vehicle localization. The solution of this invention creatively utilizes dynamic information, which was originally considered noise, in the localization process; that is, it extends the alignment from single static feature information to the joint alignment of dynamic and static feature information, improving the robustness and accuracy of the localization process.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a semantic localization method, electronic device, and machine-readable storage medium that combines dynamic and static features. Background Technology

[0002] Localization technology is one of the foundational and core technologies for robotic applications such as autonomous driving, providing robots with position and orientation information, i.e., pose information. Based on their positioning principles, existing localization technologies can be categorized into geometric localization, dead reckoning, and feature-based localization.

[0003] Geometric positioning relies on positioning facilities and is affected by signal blockage and reflection, failing in scenarios such as tunnels and overpasses. Dead reckoning has a limitation: the positioning error accumulates and increases with the estimated distance. Feature-based localization technology is widely used in robotics fields such as autonomous driving. In existing technologies, feature-based localization aligns static features detected in real-time during vehicle movement with a feature map for localization, while dynamic objects need to be filtered out. However, dynamic objects are ubiquitous in driving environments, and in certain scenarios, they even constitute the majority, such as in congested traffic, leading to a large filtering workload. In other cases, such as due to occlusion, filtering out dynamic objects is difficult, increasing computational complexity and affecting the robustness and accuracy of the localization. Summary of the Invention

[0004] In view of the above problems, a semantic localization method, electronic device and machine-readable storage medium combining dynamic and static features is proposed to overcome or at least partially solve the above problems.

[0005] One objective of this invention is to provide a semantic localization method that combines static and dynamic features, making the localization process more robust and accurate.

[0006] A further objective of this invention is to increase the road semantic information contained in the 3D observation data of the road to be registered, so as to further improve the positioning accuracy.

[0007] In particular, according to one aspect of the present invention, a semantic localization method combining dynamic and static features is provided, comprising:

[0008] The system acquires 3D sensing information of the road where the vehicle is located in real time from the sensor, processes the 3D sensing information to obtain road semantic information, which includes static semantic features and dynamic semantic features.

[0009] Based on the road semantic information, generate road 3D observation data including the static semantic features and the dynamic semantic features;

[0010] The road 3D observation data is semantically registered with a pre-constructed feature map, and the optimal registration pose is obtained through a cost function optimization method to achieve vehicle localization. The feature map includes feature vector information and road rule information. The cost function is constructed based on the matching between the static semantic features in the road 3D observation data and the feature vector information of the feature map, as well as the dynamic semantic features in the road 3D observation data and the road rule information of the feature map. The optimal registration pose minimizes the value of the cost function.

[0011] Optionally, the step of acquiring 3D sensing information of the road where the vehicle is located in real time by sensors and processing the 3D sensing information to obtain road semantic information includes:

[0012] Real-time acquisition of multi-frame 3D sensing information of the road where the vehicle is located by the sensor, and processing including segmentation and recognition of each frame of 3D sensing information to obtain multi-frame road semantic information that corresponds one-to-one with the multi-frame 3D sensing information.

[0013] The steps for generating road 3D observation data containing the static semantic features and the dynamic semantic features based on the road semantic information include:

[0014] By using a specified pose estimation algorithm, the relative poses of vehicles in multiple frames, which correspond one-to-one with the road semantic information in multiple frames, are calculated.

[0015] Based on the relative vehicle poses of the multiple frames, the road semantic information of the multiple frames is stitched together to obtain the road 3D observation data.

[0016] Optionally, the step of calculating the relative vehicle poses of multiple frames corresponding one-to-one with the road semantic information of the multiple frames by specifying a pose estimation algorithm includes:

[0017] By using a dead reckoning algorithm, the relative pose of the vehicle relative to the designated origin is calculated when the 3D sensing information of each frame is acquired, thereby obtaining the relative pose of the vehicle in multiple frames that corresponds one-to-one with the road semantic information of multiple frames.

[0018] Optionally, the step of stitching together the road semantic information from multiple frames based on the relative vehicle poses to obtain the road 3D observation data includes:

[0019] Based on the relative vehicle poses of the multiple frames, calculate the relative pose between the road semantic information of each frame and the road semantic information of the latest frame;

[0020] By using the relative poses of each frame, the road semantic information of each frame is converted to the latest frame to obtain the converted road semantic information of each frame;

[0021] The converted road semantic information from each frame is summed together to obtain the road 3D observation data.

[0022] Optionally, the step of semantically registering the road 3D observation data with a pre-constructed feature map and obtaining the optimal registration pose using a cost function optimization method includes:

[0023] Construct a static feature cost function, which is equal to the sum of the reprojection errors between each static semantic feature in the road 3D observation data after registration and pose transformation and the corresponding feature vector information in the feature map;

[0024] Multiple map regions of the road are constructed based on the road rule information of the feature map, and a dynamic feature cost function is constructed based on the dynamic semantic features of the road 3D observation data and the map regions. The multiple map regions include the drivable areas of different traffic participants and the non-drivable areas of the road other than the drivable areas. The dynamic feature cost function is equal to the sum of the matching degree between each dynamic semantic feature in the road 3D observation data and the corresponding map region after registration and pose transformation.

[0025] The sum of the static feature cost function and the dynamic feature cost function is used as the cost function to be optimized, and the optimal registration pose that minimizes the value of the cost function to be optimized is obtained.

[0026] Optionally, the degree of matching between each dynamic semantic feature in the road 3D observation data and the corresponding map region after registration and pose transformation is determined in the following way:

[0027] For each of the dynamic semantic features, if the dynamic semantic feature falls into the correct corresponding map region after registration pose transformation, then the matching degree is determined to be 1;

[0028] If the dynamic semantic features fall into the wrong corresponding map region after registration and pose transformation, then the matching degree is determined to be 0.

[0029] Optionally, the operable area includes at least one of a motor vehicle lane, a non-motor vehicle lane, and a sidewalk, and each operable area carries drivable direction information;

[0030] The non-operational area includes at least one of the following: green belts, buildings, and waterways.

[0031] Optionally, the feature map is pre-built using road information collected by high-precision positioning devices and sensors;

[0032] The feature vector information includes road surface object information and road surface marking information. The road surface objects include at least one of lampposts, road signs, curbs, and guardrails. The road surface markings include at least one of solid lines, dashed lines, arrows, and road surface text.

[0033] The road rule information includes at least one of the following: motor vehicle lanes, non-motor vehicle lanes, road direction, and road scope.

[0034] According to another aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor executes the machine-executable program to implement the aforementioned semantic localization method combining dynamic and static features.

[0035] According to another aspect of the present invention, a machine-readable storage medium is also provided, on which a machine-executable program is stored, wherein the machine-executable program, when executed by a processor, implements the aforementioned semantic localization method combining dynamic and static features.

[0036] In the semantic localization method combining dynamic and static features of this invention, localization is achieved by aligning real-time acquired dynamic and static features with a pre-constructed feature map while the vehicle is in motion. Compared with existing technologies, the solution of this invention creatively utilizes dynamic information, which was originally considered noise, in the localization process. That is, it extends the alignment from single static feature information to the joint alignment of dynamic and static feature information, thereby improving the robustness and accuracy of the localization process.

[0037] Furthermore, in the semantic localization method combining dynamic and static features of the present invention, road semantic information corresponding to multiple frames of 3D sensing information of the road is stitched together into a single frame of road 3D observation data for registration with a feature map. Compared to single-frame 3D observation data, the road 3D observation data to be registered, obtained by stitching together multiple frames of road semantic information, contains more road semantic information, making the registration more accurate and thus further improving the accuracy of the localization process.

[0038] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below.

[0039] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0040] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0041] Figure 1 A flowchart illustrating a semantic localization method combining dynamic and static features according to an embodiment of the present invention is shown.

[0042] Figure 2 A flowchart illustrating a semantic localization method combining dynamic and static features according to another embodiment of the present invention is shown.

[0043] Figure 3 A schematic diagram of a feature map according to an embodiment of the present invention is shown;

[0044] Figure 4 A schematic diagram illustrating semantic localization combining dynamic and static features according to an embodiment of the present invention is shown;

[0045] Figure 5 A schematic structural block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0046] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0047] Positioning technology is one of the foundational and core technologies for robotic applications such as autonomous driving. Based on positioning principles, positioning technology can be categorized into geometric positioning, dead reckoning, and feature-based positioning.

[0048] Geometric positioning involves measuring distance or angle using a reference device at a known location, and then determining one's own position through geometric calculations. Geometric positioning utilizes technologies such as GNSS (Global Navigation Satellite System), UWB (Ultra Wide Band), Bluetooth, and 5G to provide absolute positioning information. GNSS technology is the most widely used in intelligent vehicle applications. GNSS positioning, based on satellite positioning technology, is divided into point positioning, differential GPS positioning, and RTK (Real-time kinematic) GPS positioning. Point positioning provides 3–10 meter accuracy, differential GPS positioning provides 0.5–2 meter accuracy, and RTK GPS positioning provides centimeter-level accuracy. The limitations of geometric positioning include its reliance on positioning facilities, susceptibility to signal blockage and reflection, and failure in environments such as tunnels and overpasses.

[0049] Dead reckoning calculates the position at the next moment from the previous moment's position using motion data from sensors such as IMUs (Inertial Measurement Units) and wheel speedometers, providing relative positioning information. A limitation of dead reckoning is that the positioning error accumulates and increases with the calculated distance.

[0050] Feature-based localization first acquires several features of the surrounding environment, such as base station IDs, Wi-Fi fingerprints, images, and LiDAR point clouds. Then, it matches the observed features with a pre-established feature map to determine the location within the map, providing absolute positioning information. The direct factors affecting feature-based localization are the quantity, quality, and discriminative power of the features. The limitations of feature-based localization are that when scene and environmental factors affect feature observation, positioning accuracy and stability decrease.

[0051] The general approach to feature localization is to select static features and use them to build a feature map beforehand. During vehicle movement, the real-time detected static features are aligned with the feature map for localization. Static features (including lane markings, curbs, guardrails, buildings, etc.) are chosen for localization because they are stable and exist in the environment, while dynamic objects (including moving and stationary motor vehicles, non-motorized vehicles, pedestrians, etc.) are unstable and usually need to be filtered out to avoid interfering with localization. However, dynamic objects are ubiquitous in the driving environment, and in certain scenarios, they even constitute the majority, such as in congested traffic. In other cases, such as due to occlusion, filtering out dynamic objects can be difficult, increasing the difficulty of feature localization and affecting its robustness and accuracy.

[0052] To address, or at least partially address, the aforementioned technical problems, this invention proposes a semantic localization method that combines dynamic and static features. This method first acquires dynamic and static feature information of the road using sensors to pre-build a feature map; then, while the vehicle is in motion, the real-time acquired dynamic and static features are aligned with the feature map to achieve localization.

[0053] To facilitate the explanation of the solution, the coordinate definitions are first clarified. In this invention, a world coordinate system W is defined, which maintains a fixed relationship with the actual geographical location; for example, the Earth-Centered, Earth-Fixed coordinate system ECEF can be used. A carrier coordinate system B is also defined, which can also be called the vehicle body coordinate system for vehicles. It is fixed at a certain position on the carrier, such as the center of the rear axle. The vehicle pose, i.e., the 6Dof (Degree of Freedom) pose of the vehicle body coordinate system in the world coordinate system, is WPB = TWB. Next, a sensor coordinate system S, also called the observation coordinate system, is defined. All measurement data acquired by the sensor is based on the sensor coordinate system. Typically, the sensor is fixed to the carrier and undergoes rigid body motion with the carrier; therefore, there is a fixed transformation relationship TBS between the sensor coordinate system and the carrier coordinate system, i.e., the sensor extrinsic parameter.

[0054] Figure 1 A flowchart illustrating a semantic localization method combining dynamic and static features according to an embodiment of the present invention is shown. See also Figure 1 The method may include at least the following steps S102 to S106.

[0055] Step S102: Real-time acquisition of 3D sensing information of the road where the vehicle is located by the sensor; processing of the 3D sensing information to obtain road semantic information, which includes static semantic features and dynamic semantic features.

[0056] Step S104: Generate 3D road observation data including static semantic features and dynamic semantic features based on road semantic information;

[0057] Step S106: Semantic information registration is performed between the road 3D observation data and the pre-constructed feature map. The optimal registration pose is obtained through the cost function optimization method to achieve vehicle localization. The feature map includes feature vector information and road rule information. The cost function is constructed based on the matching between the static semantic features in the road 3D observation data and the feature vector information of the feature map, as well as the dynamic semantic features in the road 3D observation data and the road rule information of the feature map. The optimal registration pose minimizes the value of the cost function.

[0058] In the semantic localization method combining dynamic and static features provided in this invention, localization is achieved by aligning real-time acquired dynamic and static features with a pre-constructed feature map while the vehicle is in motion. Compared with existing technologies, the present invention creatively utilizes dynamic information, which was originally considered noise, in the localization process. That is, it expands from aligning single static feature information to aligning both dynamic and static feature information, thereby improving the robustness and accuracy of the localization process.

[0059] In embodiments of the present invention, a feature map needs to be pre-built for alignment with real-time acquired dynamic and static features to achieve vehicle positioning. In some embodiments, the feature map is pre-built using road information collected by high-precision positioning devices and sensors. The sensors can be single sensors or combinations of multiple sensors, such as cameras, LiDAR (Laser Detection and Ranging), or other sensors.

[0060] The feature map includes feature vector information and road rule information. Feature vector information stores road features in a vector format, including but not limited to storing information about at least one of the road surface objects such as light poles, road signs, curbs, and guardrails using vector information such as points, lines, and areas, as well as storing information about at least one of the road markings such as solid lines, dashed lines, arrows, and text. Road rule information includes but is not limited to at least one of the following: motor vehicle lanes, non-motor vehicle lanes, road direction, and road extent.

[0061] In embodiments of the present invention, 3D sensing information of the road is acquired in real time through sensors equipped on the vehicle. These sensors may be, for example, cameras, LiDAR, or other sensors, or combinations thereof. The 3D sensing information of the road is acquired through these sensors. Then, road semantic information is obtained through detection, segmentation, and recognition methods. Existing recognition technologies can be used for the recognition method. Road semantic information includes static semantic features and dynamic semantic features. Specifically, static semantic features may include stable road surface objects and road markings such as lane markings, curbs, guardrails, and buildings. Dynamic semantic features may include unstable dynamic objects such as moving and stationary motor vehicles, non-motorized vehicles, and pedestrians.

[0062] Furthermore, road 3D observation data, including static and dynamic semantic features, are generated based on road semantic information.

[0063] As mentioned above, those skilled in the art should recognize that, due to the fixed relationship between the sensor coordinate system and the vehicle coordinate system, before using the real-time obtained road semantic information for alignment with the feature map, the road semantic information obtained from the 3D sensing information of the sensor should be transformed into the vehicle coordinate system through sensor extrinsic parameters, thereby generating road 3D observation data in the vehicle coordinate system. Specifically, let the road semantic information (also called observation information) obtained from the 3D sensing information of sensor A be PA, and the sensor extrinsic parameter be TBA, then the road semantic information in the vehicle coordinate system is PB = TBA * PA.

[0064] In embodiments of the present invention, road 3D sensing data for registration can be obtained based on 3D sensing information of a single frame, or road 3D sensing data for registration can be obtained based on 3D sensing information of multiple frames.

[0065] In a preferred embodiment, road 3D sensing data for registration is obtained based on multi-frame 3D sensing information. Specifically, multi-frame 3D sensing information of the road where the vehicle is located is acquired in real time by the sensor. Each frame of 3D sensing information is processed to obtain multi-frame road semantic information that corresponds one-to-one with the multi-frame 3D sensing information. The processing of each frame of 3D sensing information may include segmentation and recognition, and may also include necessary detection. Furthermore, the road semantic information of each frame can also be transformed into the vehicle coordinate system using sensor extrinsic parameters.

[0066] Then, using a specified pose estimation algorithm, multi-frame relative vehicle poses corresponding one-to-one with multi-frame road semantic information are calculated. Based on these multi-frame relative vehicle poses, the multi-frame road semantic information is then stitched together to obtain 3D road observation data. In practical applications, any algorithm capable of estimating vehicle poses can be used for the specified pose estimation algorithm. In this embodiment, the road semantic information corresponding to multi-frame 3D sensing information is stitched together into a single frame of 3D road observation data for subsequent registration with the feature map. Compared to single-frame 3D observation data, this 3D road observation data to be registered, obtained by stitching together multi-frame road semantic information, contains more road semantic information, making the registration more accurate and further improving the accuracy of the localization process.

[0067] In a preferred embodiment, dead reckoning algorithms can be used to calculate the vehicle's relative pose. In this case, sensors such as IMUs, wheel speedometers, or vehicle speedometers need to be installed on the vehicle to acquire the motion data required for dead reckoning. Specifically, the dead reckoning algorithm calculates the vehicle's relative pose relative to a specified origin when acquiring each frame of 3D sensing information, thus obtaining multi-frame vehicle relative poses that correspond one-to-one with multi-frame road semantic information. The vehicle's relative pose obtained through dead reckoning (DR) refers to the relative pose from point a to point b in the DR coordinate system provided by the DR. The DR coordinate system is defined by the DR, and generally, the pose when the DR acquires the first frame of observation data can be taken as the origin. Specifically, let the pose of point a be Ta and the pose of point b be Tb, then the relative pose between points a and b is Tba = Ta - inverse * Tb, where Ta - inverse refers to the inverse matrix of Ta. Using dead reckoning algorithms can obtain more stable vehicle relative pose data.

[0068] Furthermore, by using the aforementioned relative vehicle poses, multiple frames of road semantic information are stitched together to obtain 3D road observation data. Specifically, the relative poses between the road semantic information of each frame and the road semantic information of the latest frame are calculated; using each relative pose, the road semantic information of each frame is transformed to the latest frame to obtain transformed road semantic information of each frame; and all the transformed road semantic information of each frame is accumulated together to obtain 3D road observation data. Let the multi-frame road semantic information be represented as F1, F2…Fn, and the corresponding DR vehicle relative poses be T1, T2…Tn, where Fn is the latest frame road semantic information. The above calculation process can be expressed as follows: First, calculate the relative pose of each frame road semantic information with the latest frame road semantic information. For the i-th frame road semantic information, its relative pose Tni = Ti - inverse * Tn, where Ti - inverse is the inverse matrix of Ti. Then, through the relative pose of each frame road semantic information with the latest frame road semantic information, transform each frame road semantic information to the latest frame. For the i-th frame road semantic information, its transformed road semantic information nFi = Tni * Fi. Finally, directly accumulate all the transformed road semantic information to the latest frame to obtain the stitched road 3D observation data.

[0069] After obtaining the 3D road observation data, the 3D road observation data is semantically registered with the corresponding feature map to obtain the optimal registration pose (TWB). The cost function optimization method is used to obtain the registration pose (TWB). The cost function to be optimized includes two parts: static features and dynamic features.

[0070] In a specific embodiment, semantic information registration is performed between road 3D observation data and a pre-constructed feature map, and the optimal registration pose is obtained through a cost function optimization method. The steps of obtaining the optimal registration pose through the cost function optimization method may include:

[0071] Construct a static feature cost function, which is equal to the sum of the reprojection errors between each static semantic feature in the road 3D observation data after registration and pose transformation and the corresponding feature vector information in the feature map;

[0072] Multiple map regions of a road are constructed based on the road rule information of the feature map, and a dynamic feature cost function is constructed based on the dynamic semantic features of the road 3D observation data and the map regions. The multiple map regions include the drivable areas of different traffic participants and the non-drivable areas in the road other than the drivable areas. The dynamic feature cost function is equal to the sum of the matching degree between each dynamic semantic feature in the road 3D observation data and the corresponding map region after registration and pose transformation.

[0073] The sum of the static feature cost function and the dynamic feature cost function is used as the cost function to be optimized, and the optimal registration pose that minimizes the value of the cost function to be optimized is obtained.

[0074] In the above steps, the navigable areas for different traffic participants may include at least one of the following: motor vehicle lanes, non-motor vehicle lanes, and sidewalks, and each navigable area carries directional information. Non-navigable areas may include at least one of the following in the road environment: green belts, buildings, and waterways.

[0075] Furthermore, the matching degree between each dynamic semantic feature in the road 3D observation data and its corresponding map region after registration pose transformation is determined as follows: For each dynamic semantic feature, if the dynamic semantic feature falls into the correct corresponding map region after registration pose transformation, the matching degree is determined to be 1; if the dynamic semantic feature falls into the wrong corresponding map region after registration pose transformation, the matching degree is determined to be 0. For example, for a dynamic semantic feature identified as a motor vehicle, if it falls into the motor vehicle lane after registration pose TWB transformation, its matching degree is determined to be 1; if it falls into a non-operational area such as a sidewalk or green belt after TWB transformation, its matching degree is determined to be 0.

[0076] The above introduces Figure 1 The embodiments shown have various implementation methods for each stage. The following will describe in detail the implementation process of the semantic localization method combining dynamic and static features of the present invention through specific embodiments.

[0077] Figure 2A flowchart illustrating a semantic localization method combining dynamic and static features according to a specific embodiment of the present invention is shown. In this embodiment, road information is acquired in advance using high-precision positioning equipment and sensors, and a feature map is pre-built. During the feature map construction stage, feature vector information and road rule information are stored in the feature map respectively. Figure 3 This is a schematic diagram of a feature map. In the illustrated feature map, feature vector information includes information about road surface objects such as curbs, as well as road markings such as solid lines, dashed lines, arrows, stop lines, zebra crossings, and text. Road rule information includes information such as motor vehicle lanes, non-motor vehicle lanes, and road direction.

[0078] like Figure 2 As shown, the semantic localization method combining dynamic and static features may include the following steps S202 to S216.

[0079] Step S202: Real-time acquisition of multi-frame 3D sensing information of the road where the vehicle is located by the sensor, processing of each frame of 3D sensing information including segmentation and recognition, to obtain multi-frame road semantic information that corresponds one-to-one with the multi-frame 3D sensing information.

[0080] Vehicles are equipped with cameras, LiDAR, or other sensors, or combinations thereof. These sensors acquire 3D sensing information about the road. Then, through detection, segmentation, and recognition methods, road semantic information is obtained. Road semantic information includes static semantic features and dynamic semantic features.

[0081] By using sensor extrinsic parameters, this road semantic information is transformed into the vehicle coordinate system.

[0082] The multi-frame road semantic information obtained in this step is represented as F1, F2…Fn, where Fn is the road semantic information of the latest frame.

[0083] Step S204: Using a dead reckoning algorithm, calculate the relative pose of the vehicle relative to the vehicle pose at the specified origin when acquiring each frame of 3D sensing information, thereby obtaining the relative poses of the vehicle in multiple frames that correspond one-to-one with the road semantic information in multiple frames.

[0084] The vehicle is equipped with sensors such as an inertial measurement unit, wheel speedometer, or vehicle speedometer. The relative pose of the vehicle in each frame of 3D sensing information is obtained through dead reckoning calculation. Here, relative pose reckoning refers to the relative pose from point a to point b provided by the dead reckoning (DR). Specifically, in the DR coordinate system, if the pose of point a is Ta and the pose of point b is Tb, then the relative pose between a and b can be obtained as Tba = Ta - inverse * Tb, where Ta - inverse refers to the inverse matrix of Ta.

[0085] The relative poses of the DR vehicles obtained in this step, which correspond one-to-one with the road semantic information of multiple frames, are represented as T1, T2…Tn.

[0086] Step S206: Based on the relative poses of the vehicles in the multiple frames, calculate the relative poses between the road semantic information of each frame and the road semantic information of the latest frame.

[0087] For the road semantic information of the i-th frame, its relative pose Tni with the road semantic information of the latest frame is Tni = Ti - inverse * Tn, where Ti - inverse is the inverse matrix of Ti.

[0088] Step S208: Convert the road semantic information of each frame to the latest frame through each relative pose to obtain the converted road semantic information of each frame.

[0089] For the road semantic information in the i-th frame, the road semantic information nFi after conversion to the latest frame is Tni*Fi.

[0090] Step S210: All the converted road semantic information from each frame is accumulated together to obtain road 3D observation data.

[0091] In this step, all road semantic information converted to the latest frame is directly accumulated to obtain the stitched 3D road observation data.

[0092] Step S212: Construct the static feature cost function.

[0093] The static feature cost function is equal to the sum of the reprojection errors between each static semantic feature in the road 3D observation data after registration and pose transformation and the corresponding feature vector information in the feature map. Expressed as a formula, assuming the static semantic features in the road 3D observation data are S1, S2, S3…Sn, and the corresponding feature vector information in the feature map is M1, M2, M3…Mm, then the static feature cost function is:

[0094] F-static(TWB)=SUM(DistStatic(TWB*Si,Mi))

[0095] Where DistStatic(*) represents the reprojection error between the static semantic features (also known as static observation elements) Si and the feature vector information (also known as map elements) Mi of the feature map, and SUM(*) represents the sum of the reprojection errors of all static semantic features and their corresponding feature vector information. m and n can be equal or unequal. Generally speaking, due to environmental factors, occlusion, etc., the number of static observation elements acquired in real time may be less than the number of elements in the feature map; therefore, m ≥ n.

[0096] Step S214: Construct multiple map regions of the road based on the road rule information of the feature map, and construct a dynamic feature cost function based on the dynamic semantic features of the road 3D observation data and the map regions.

[0097] In this step, road rule information from the feature map is obtained, and navigable areas Z1, Z2, Z3...Zg for different traffic participants are constructed, including but not limited to: motor vehicle lanes, non-motor vehicle lanes, and sidewalks. Each navigable area also includes directional information. Areas outside the navigable areas are classified as non-navigable areas Zx, corresponding to non-navigable areas in the road environment such as green belts, buildings, and waterways.

[0098] Assuming the dynamic semantic features in the 3D road observation data are D1, D2, D3...Dk, then the dynamic feature cost function is:

[0099] F-dynamic(TWB)=SUM(DistZone(TWB*Di,Zi,Zx))

[0100] Where DistZone(*) represents the degree of matching between the dynamic semantic feature (also known as the dynamic observation element) Di and the corresponding map region Zi / Zx, and SUM(*) represents the sum of the degree of matching between all dynamic semantic features and the corresponding map regions. There are no restrictions on the relative magnitudes of g and k.

[0101] Furthermore, DistZone(*) can be represented as:

[0102] DistZone(TWB*Di,Zi,Zx) = 1 if TWB*Di is in the correct region Zi.

[0103] =0 if TWB*Di is in the wrong region Zj or Zx.

[0104] That is, if the dynamic semantic feature falls into the correct corresponding map region after TWB transformation, the matching score is 1; if the dynamic semantic feature falls into the wrong corresponding map region after TWB transformation, the matching score is 0.

[0105] Step S216: The sum of the static feature cost function and the dynamic feature cost function is used as the cost function to be optimized, and the optimal registration pose that minimizes the value of the cost function to be optimized is obtained, thereby realizing the vehicle localization.

[0106] The cost function to be optimized is expressed as:

[0107] F(TWB)=F-static(TWB)+F-dynamic(TWB)

[0108] The optimized solution can be expressed as:

[0109] TWB = argmin(F(TWB))

[0110] Here, argmin(*) represents finding the optimal TWB that minimizes the cost function. This allows for the acquisition of the vehicle's high-precision pose through registration.

[0111] The order of steps S212 and S214 can be interchanged, or they can be performed simultaneously.

[0112] Figure 4 The diagram illustrates semantic localization based on a combination of dynamic and static features according to this embodiment. Figure 4 As shown, after registration, in addition to the alignment of static semantic features with feature vector information, dynamic semantic features also fall into the corresponding map areas. For example, pedestrians fall into zebra crossings, non-motorized vehicles fall into non-motorized vehicle lanes, and motorized vehicles fall into motorized vehicle lanes. This achieves high-precision positioning of the vehicle.

[0113] This embodiment acquires dynamic and static feature information through multiple sensors and aligns it with a pre-established feature map to achieve localization, making the localization process more robust and accurate. During the map construction phase, feature vector information and road rule information are stored in the map to obtain a feature map that can be aligned with the dynamic and static information. Furthermore, dead reckoning technology is used to obtain the relative pose information of the vehicle, and multiple frames of dynamic and static features are stitched together to obtain 3D road observation data, making subsequent localization more reliable and robust.

[0114] Based on the same inventive concept, embodiments of the present invention also provide an electronic device 200. See also Figure 5 As shown, the electronic device 200 includes a memory 201, a processor 202, and a machine-executable program 203 stored in the memory 201 and running on the processor 202. When the processor 202 executes the machine-executable program 203, it implements the semantic localization method combining dynamic and static features of any of the foregoing embodiments or combinations of embodiments.

[0115] Based on the same inventive concept, embodiments of the present invention also provide a machine-readable storage medium. This machine-readable storage medium stores a machine-executable program, which, when executed by a processor, implements the semantic localization method combining dynamic and static features of any of the foregoing embodiments or combinations thereof.

[0116] Those skilled in the art will clearly understand that the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and for the sake of brevity, it will not be repeated here.

[0117] Furthermore, the functional units in the various embodiments of the present invention can be physically independent of each other, or two or more functional units can be integrated together, or all functional units can be integrated into one processing unit. The integrated functional units described above can be implemented in hardware, or in software or firmware.

[0118] Those skilled in the art will understand that if the integrated functional unit is implemented in software and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of the present invention when running the instructions. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as a computing device, personal computer, server, or network device) related to program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the computing device, the computing device executes all or part of the steps of the methods described in the various embodiments of the present invention.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to depart from the protection scope of the present invention.

Claims

1. A semantic localization method combining dynamic and static features, comprising: The system acquires 3D sensing information of the road where the vehicle is located in real time from the sensor, processes the 3D sensing information to obtain road semantic information, which includes static semantic features and dynamic semantic features. Based on the road semantic information, generate road 3D observation data including the static semantic features and the dynamic semantic features; The road 3D observation data is semantically registered with a pre-constructed feature map, and the optimal registration pose is obtained through a cost function optimization method to achieve vehicle localization. The feature map includes feature vector information and road rule information. The cost function is constructed based on the matching between the static semantic features in the road 3D observation data and the feature vector information of the feature map, as well as the dynamic semantic features in the road 3D observation data and the road rule information of the feature map. The optimal registration pose minimizes the value of the cost function. The steps of semantically registering the road 3D observation data with a pre-constructed feature map and obtaining the optimal registration pose using a cost function optimization method include: Construct a static feature cost function, which is equal to the sum of the reprojection errors between each static semantic feature in the road 3D observation data after registration and pose transformation and the corresponding feature vector information in the feature map; Multiple map regions of the road are constructed based on the road rule information of the feature map, and a dynamic feature cost function is constructed based on the dynamic semantic features of the road 3D observation data and the map regions. The multiple map regions include the drivable areas of different traffic participants and the non-drivable areas of the road other than the drivable areas. The dynamic feature cost function is equal to the sum of the matching degree between each dynamic semantic feature in the road 3D observation data and the corresponding map region after registration and pose transformation. The sum of the static feature cost function and the dynamic feature cost function is used as the cost function to be optimized, and the optimal registration pose that minimizes the value of the cost function to be optimized is obtained.

2. The semantic positioning method of claim 1, wherein, The steps of acquiring 3D sensing information of the road where the vehicle is located in real time by sensors and processing the 3D sensing information to obtain road semantic information include: Real-time acquisition of multi-frame 3D sensing information of the road where the vehicle is located by the sensor, and processing including segmentation and recognition of each frame of 3D sensing information to obtain multi-frame road semantic information that corresponds one-to-one with the multi-frame 3D sensing information. The steps for generating road 3D observation data containing the static semantic features and the dynamic semantic features based on the road semantic information include: By using a specified pose estimation algorithm, the relative poses of vehicles in multiple frames, which correspond one-to-one with the road semantic information in multiple frames, are calculated. Based on the relative vehicle poses of the multiple frames, the road semantic information of the multiple frames is stitched together to obtain the road 3D observation data.

3. The semantic localization method according to claim 2, wherein, The steps for calculating the relative vehicle poses of multiple frames, which correspond one-to-one with the road semantic information of the multiple frames, by using a specified pose estimation algorithm include: By using a dead reckoning algorithm, the relative pose of the vehicle relative to the designated origin is calculated when the 3D sensing information of each frame is acquired, thereby obtaining the relative pose of the vehicle in multiple frames that corresponds one-to-one with the road semantic information of multiple frames.

4. The semantic localization method according to claim 2, wherein, The steps of stitching together the road semantic information from multiple frames to obtain the road 3D observation data based on the relative vehicle poses of the multiple frames include: Based on the relative vehicle poses of the multiple frames, calculate the relative pose between the road semantic information of each frame and the road semantic information of the latest frame; By using the relative poses of each frame, the road semantic information of each frame is converted to the latest frame to obtain the converted road semantic information of each frame; The converted road semantic information from each frame is summed together to obtain the road 3D observation data.

5. The semantic localization method according to any one of claims 1-4, wherein, The degree of matching between each dynamic semantic feature in the road 3D observation data and the corresponding map region after registration and pose transformation is determined by the following method: For each of the dynamic semantic features, if the dynamic semantic feature falls into the correct corresponding map region after registration pose transformation, then the matching degree is determined to be 1; If the dynamic semantic features fall into the wrong corresponding map region after registration and pose transformation, then the matching degree is determined to be 0.

6. The semantic localization method according to any one of claims 1-4, wherein, The operable area includes at least one of a motor vehicle lane, a non-motor vehicle lane, and a sidewalk, and each operable area carries drivable direction information; The non-operational area includes at least one of the following: green belts, buildings, and waterways.

7. The semantic localization method according to claim 1, wherein, The feature map is pre-constructed by collecting road information using high-precision positioning equipment and sensors; The feature vector information includes road surface object information and road surface marking information. The road surface objects include at least one of lampposts, road signs, curbs, and guardrails. The road surface markings include at least one of solid lines, dashed lines, arrows, and road surface text. The road rule information includes at least one of the following: motor vehicle lanes, non-motor vehicle lanes, road direction, and road scope.

8. An electronic device comprising a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor, when executing the machine-executable program, implements a semantic localization method combining dynamic and static features according to any one of claims 1-7.

9. A machine-readable storage medium having a machine-executable program stored thereon, the machine-executable program, when executed by a processor, implementing the semantic localization method combining dynamic and static features according to any one of claims 1-7.

Images