Construction method of three-dimensional semantic map, vehicle, storage medium and program product

Abstract

The invention discloses a construction method of a three-dimensional semantic map, a vehicle, a storage medium and a program product. The construction method of the three-dimensional semantic map comprises the steps of obtaining time sequence image data collected by image collection devices of at least two visual angles, and obtaining pose data of the vehicle provided with the image collection devices; obtaining a three-dimensional aerial view feature map based on time sequence image data acquired by image acquisition devices of at least two visual angles; obtaining a three-dimensional point cloud frame based on the time sequence image data collected by the image collection device of at least two visual angles and the pose data of the vehicle; and constructing a three-dimensional semantic graph based on the three-dimensional aerial view feature map and the three-dimensional point cloud frame. According to the method, the problems that a semantic map constructed in the prior art is low in map matching efficiency and poor in vehicle positioning robustness can be solved.

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and specifically relates to a method and vehicle for constructing a three-dimensional semantic map, as well as a storage medium and a program product. Background Art

[0002] In vehicle autonomous driving technology, accurate vehicle positioning and navigation are important factors to ensure safe driving of the vehicle, especially when the vehicle is driving in dense urban scenes or complex roads, the accuracy of vehicle positioning and navigation information is required to be higher to ensure driving safety. In the process of implementing this application, the inventor found that the semantic map constructed by the prior art has technical problems of low map matching efficiency and poor vehicle positioning robustness. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a method and vehicle for constructing a three-dimensional semantic map, as well as an electronic device, a computer-readable storage medium, and a computer program product, which can solve the problems of low map matching efficiency and poor vehicle positioning robustness in the semantic map constructed by the prior art.

[0004] In a first aspect, an embodiment of the present application provides a method for constructing a three-dimensional semantic map, comprising:

[0005] Acquire time-series image data acquired by image acquisition devices of at least two viewing angles, and acquire position and posture data of a vehicle on which the image acquisition devices are installed;

[0006] Obtaining a three-dimensional bird's-eye view feature map based on time-series image data collected by image acquisition devices of at least two viewing angles;

[0007] Obtaining a three-dimensional point cloud frame based on time-series image data collected by image acquisition devices of at least two viewing angles and position data of the vehicle;

[0008] A 3D semantic map is constructed based on the 3D bird's-eye view feature map and 3D point cloud frame.

[0009] In some embodiments, the three-dimensional semantic map is constructed based on the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame, including:

[0010] A 3D semantic map is constructed based on the 3D bird's-eye view feature map, 3D point cloud frame and vehicle posture data.

[0011] In some embodiments, the three-dimensional bird's-eye view feature map is obtained by using the time-series image data collected by the image collection device based on at least two viewing angles, including:

[0012] Acquiring two-dimensional image features based on time-series image data;

[0013] Acquire image depth estimation information of two-dimensional image features based on device parameter information of the image acquisition device;

[0014] A three-dimensional bird's-eye view feature map is generated based on the two-dimensional image features and image depth estimation information of the two-dimensional image features.

[0015] In some embodiments, the three-dimensional semantic map is constructed based on the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame and the vehicle's posture data, including:

[0016] Performing information recognition on the three-dimensional bird's-eye view feature map to obtain first map information in the three-dimensional bird's-eye view feature map;

[0017] A three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame, and the vehicle's posture data.

[0018] In some embodiments, the above-mentioned information recognition of the three-dimensional bird's-eye view feature map to obtain the first map information in the three-dimensional bird's-eye view feature map includes at least one of the following:

[0019] Performing scene semantic segmentation on the three-dimensional bird's-eye view feature map to obtain first map element information in the three-dimensional bird's-eye view feature map;

[0020] Target detection is performed on the 3D bird's-eye view feature map to obtain the motion element information in the 3D bird's-eye view feature map.

[0021] In some embodiments, the three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame, and the vehicle's position data, including:

[0022] A three-dimensional semantic map is constructed based on the first map element information, motion element information and vehicle posture data in the three-dimensional bird's-eye view feature map.

[0023] In some embodiments, the three-dimensional point cloud frame is obtained by using the time-series image data and the vehicle posture data collected by the image acquisition device based on at least two viewing angles, including:

[0024] Extracting feature points based on time-series image data collected by image collection devices of at least two viewing angles to obtain first feature point information;

[0025] Obtaining a three-dimensional point cloud frame based on the first feature point information and the position and posture data of the vehicle;

[0026] Information recognition is performed on the three-dimensional point cloud frame to obtain second map element information in the three-dimensional point cloud frame.

[0027] In some embodiments, after obtaining the three-dimensional point cloud frame, the method further includes:

[0028] Optimizing the point cloud coordinates in the three-dimensional point cloud frame based on the global optimization function to obtain an optimized three-dimensional point cloud frame;

[0029] Performing information recognition on the three-dimensional point cloud frame includes:

[0030] Information recognition is performed on the optimized three-dimensional point cloud frame to obtain second map element information in the three-dimensional point cloud frame.

[0031] In some embodiments, the three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame, and the vehicle's posture data, including:

[0032] A three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the second map element information in the three-dimensional point cloud frame, and the vehicle's posture data.

[0033] In some embodiments, the three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the second map element information in the three-dimensional point cloud frame, and the vehicle's position data, including:

[0034] Based on the first map information in the three-dimensional bird's-eye view feature map and the second map element information in the three-dimensional point cloud frame, a fusion process is performed to obtain fused map element information and form a plurality of fused feature maps;

[0035] Extract key frames from multiple fused feature maps;

[0036] Based on the fused map element information included in the key frames and the vehicle's position data, a local 3D semantic map is constructed;

[0037] Based on the vehicle's posture data, the local 3D semantic maps are merged to obtain a global 3D semantic map.

[0038] In some embodiments, the above-mentioned construction of a local three-dimensional semantic map based on the fused map element information included in the key frame and the vehicle's position data includes:

[0039] The vehicle coordinate system at the initial moment is obtained based on the vehicle's posture data and used as the map coordinate system;

[0040] Based on the map coordinate system and the fused map element information included in the key frames, a local three-dimensional semantic map is constructed.

[0041] In some embodiments, after constructing the local three-dimensional semantic map based on the map coordinate system and the fused map element information included in the key frame, the following further includes:

[0042] The coordinate values ​​of the fused map element information in the local 3D semantic map are converted into coordinate values ​​in the global coordinate system.

[0043] In some embodiments, the above-mentioned merging process of the local 3D semantic map based on the vehicle's position data to obtain the global 3D semantic map includes:

[0044] Generate vehicle motion trajectory based on vehicle posture data;

[0045] The local 3D semantic maps are merged based on the vehicle motion trajectory to obtain a global 3D semantic map.

[0046] In some embodiments, the global three-dimensional semantic map is stored in a tree data structure, and the fused map element information of the local three-dimensional semantic map is stored in the nodes of the tree data structure.

[0047] In some embodiments, the above-mentioned merging process of the local three-dimensional semantic map based on the vehicle motion trajectory includes:

[0048] Determine whether the key frame for constructing the local three-dimensional semantic map includes a loop candidate frame;

[0049] When the local 3D semantic map includes loop candidate frames, the pose of the local 3D semantic map is globally optimized based on the constructed global 3D semantic map;

[0050] The local three-dimensional semantic maps after global optimization are merged based on the vehicle motion trajectory.

[0051] In some embodiments, the above-mentioned determination of whether the key frame for constructing the local three-dimensional semantic map includes a loop candidate frame includes:

[0052] Whether a key frame for constructing a local three-dimensional semantic map includes a loop candidate frame is determined by at least one of global coordinate matching, bag-of-words model, or feature database matching.

[0053] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method for constructing a three-dimensional semantic map as described in the first aspect are implemented.

[0054] In a third aspect, an embodiment of the present application provides a vehicle, comprising the electronic device described in the fifth aspect, wherein a memory of the electronic device stores programs or instructions that can be run on a processor, and when the program or instructions are executed by the processor, the steps of the method for constructing a three-dimensional semantic map as described in the first aspect are implemented.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for constructing a three-dimensional semantic map as described in the first aspect are implemented.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed by a processor, implements the steps of the method for constructing a three-dimensional semantic map as described in the first aspect.

[0057] The technical solution provided in the present application is to construct a three-dimensional semantic map based on a three-dimensional bird's-eye view feature map and a three-dimensional point cloud frame. It can use the information in the multi-perspective time-series image data to obtain a three-dimensional bird's-eye view feature map and a three-dimensional point cloud frame, and then fuse the map element information in the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame. Then, the three-dimensional semantic map is constructed based on the richer and more accurate map element information, wherein the map element information in the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame can be complementary and associated, and the map element data in the three-dimensional semantic map is time-optimized, thereby improving the map matching efficiency and the robustness of positioning during the vehicle positioning process.

[0058] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0060] Figure 1 A schematic diagram of a process for constructing a three-dimensional semantic map in an embodiment of the present application;

[0061] Figure 2 for Figure 1 A schematic diagram of a specific execution flow of step 102 of the illustrated embodiment;

[0062] Figure 3 This is another schematic diagram of a process for constructing a three-dimensional semantic map in an embodiment of the present application;

[0063] Figure 4 This is a schematic diagram of a process for obtaining second map element information in a three-dimensional point cloud frame in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of another process of constructing a three-dimensional semantic map in an embodiment of the present application;

[0065] Figure 6 for Figure 5A schematic diagram of a specific execution flow of step 503 of the illustrated embodiment;

[0066] Figure 7 A schematic diagram of a process of obtaining a global three-dimensional semantic map based on a local three-dimensional semantic map in an embodiment of the present application;

[0067] Figure 8 A schematic diagram of a process of merging local three-dimensional semantic maps in an embodiment of the present application;

[0068] Fig. 9 A schematic diagram of a vehicle positioning method according to an embodiment of the present application;

[0069] Fig.10 A schematic diagram of a vehicle navigation method according to an embodiment of the present application;

[0070] Fig.11 A schematic diagram of the structure of a device for constructing a three-dimensional semantic map in an example of this application;

[0071] Fig.12 This is a schematic diagram of the structure of a vehicle positioning device in an example of this application;

[0072] Fig.13 This is a schematic diagram of the structure of a vehicle navigation device in an example of this application;

[0073] Fig.14 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0074] Fig.15 It is a schematic diagram of the structure of a vehicle in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0076] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0077] In some related technologies, in the process of constructing a three-dimensional semantic map using existing technologies, when performing template matching between a three-dimensional semantic local map and a three-dimensional semantic global map, it fails to introduce temporal data information in the matching stage, and fails to perform temporal optimization processing on the target data in the constructed three-dimensional semantic local map. This will result in the constructed three-dimensional semantic map having problems of low map matching efficiency and poor vehicle positioning robustness.

[0078] In addition, the above-mentioned 3D semantic construction technology usually uses image data acquired by a color depth camera to construct a 3D semantic map. However, the color depth camera is limited by factors such as lighting conditions and power, which makes it impossible to use it in outdoor scenes or high-speed motion scenes. In addition, there is a fusion blind spot between the scene representation of its depth map and the result of image semantic segmentation. When using image data collected by a color depth camera with a single modality and a single perspective to construct a 3D semantic map, it does not fully consider the failure of image features in weak texture areas and medium and high-speed scenes, and only reconstructs the scene based on static map elements. It does not perform vectorized abstraction of map elements and constraints on the association relationships between elements, thereby increasing the risk of positioning failure in the vehicle's positioning and matching stage.

[0079] Therefore, the technical solution provided by the present application constructs a three-dimensional semantic map based on a three-dimensional bird's-eye view feature map and a three-dimensional point cloud frame. It can utilize the information in the multi-perspective time-series image data to obtain a three-dimensional bird's-eye view feature map and a three-dimensional point cloud frame, and then fuse the map element information in the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame. Then, the three-dimensional semantic map is constructed based on the richer and more accurate map element information, wherein the map element information in the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame can be complementary and associated, and the map element data in the three-dimensional semantic map can be time-optimized, thereby improving the map matching efficiency and the robustness of positioning during the vehicle positioning process.

[0080] Figure 1 FIG. 1 is a flow chart of a method for constructing a three-dimensional semantic map in an embodiment of the present application. The vehicle braking control method can be implemented by a corresponding three-dimensional semantic map construction device, such as Figure 1 As shown, the method comprises the following steps:

[0081] Step 101, acquiring time-series image data acquired by image acquisition devices of at least two viewing angles, and acquiring position and posture data of a vehicle on which the image acquisition devices are installed;

[0082] The time-series image data in the technical solution provided in the embodiment of the present application is time-synchronized, multi-view image data acquired by the image acquisition device, and multiple time-series image data of the same view are arranged in chronological order and have continuity. The posture data includes the position information and posture information of the vehicle, which is helpful to understand the motion state of the vehicle. The above position information includes the longitude, latitude, altitude and other information of the vehicle; the posture information may include the roll angle, pitch angle, heading angle and other information of the vehicle.

[0083] Step 102, obtaining a three-dimensional bird's-eye view feature map based on time-series image data collected by image collection devices of at least two viewing angles;

[0084] On the basis of step 101, a three-dimensional bird's-eye view feature map is obtained based on the above-mentioned time-series image data. The three-dimensional bird's-eye view feature map includes feature information such as position information, depth information, and semantic information of objects in the time-series image data. The three-dimensional bird's-eye view feature map can be obtained based on the time-series image data and using a bottom-up or top-down method. The process of obtaining the three-dimensional bird's-eye view feature map using the above-mentioned bottom-up method is: based on the acquired 2D time-series image data, through a specific algorithm, for example, through operations such as Lift, Splat, and Shoot in the LSS framework, the image features of the above-mentioned 2D time-series image data are converted into a three-dimensional point cloud, which can also be called a 3D point cloud, and then the three-dimensional bird's-eye view feature map is generated using techniques such as voxel pooling. Common bottom-up methods include LSS, BEVDet, BEVDepth, etc.; the process of obtaining the three-dimensional bird's-eye view feature map using the above-mentioned top-down method is: first predefine the three-dimensional bird's-eye view feature map to be generated, and then use global perception capabilities such as Transformer to query the corresponding information multiple times in the features of multiple perspective images, and fuse and update the information into the BEV feature map.

[0085] In this step, the three-dimensional bird's-eye view feature map can be obtained by converting the perspective view of the time series image data into a bird's-eye view, utilizing the corresponding depth information, and projecting the bird's-eye view into 3D coordinates.

[0086] Step 103, obtaining a three-dimensional point cloud frame based on the time-series image data collected by the image acquisition device of at least two viewing angles and the position and posture data of the vehicle;

[0087] This step is to obtain a 3D point cloud frame based on these data and a corresponding algorithm after obtaining the time-series image data of at least two perspectives and the vehicle's posture data in step 101. The 3D point cloud frame is a set of discrete points in a 3D space. The above-mentioned algorithms include algorithms such as motion recovery structure, depth estimation and structure reconstruction based on deep learning, and 3D reconstruction based on a color depth camera. In the embodiment of the present application, the 3D point cloud frame can be a sparse 3D point cloud frame.

[0088] Step 104: construct a three-dimensional semantic map based on the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame.

[0089] This step is to construct a 3D semantic map based on the 3D bird's-eye view feature map and the 3D point cloud frame after obtaining the 3D bird's-eye view feature map and the 3D point cloud frame in step 102 and step 103, that is, first fuse the two, and then use the fusion result to construct a 3D semantic map. The 3D semantic map includes regional scene information, attribute information, posture information, elevation direction information, etc. of each independent object in the scene.

[0090] As described above, the three-dimensional semantic map constructed by this method can not only utilize the information in the multi-perspective temporal image data to complement and associate the map element information, but also perform temporal optimization processing on the map element data in the three-dimensional semantic map, thereby improving the map matching efficiency and positioning robustness of the vehicle during the positioning process.

[0091] In some embodiments, a 3D semantic map can be constructed based on a 3D bird's-eye view feature map and a 3D point cloud frame. Specifically, a 3D semantic map can be constructed using the 3D bird's-eye view feature map, the 3D point cloud frame, and the vehicle's position data. The 3D semantic map constructed based on the above data can not only better reflect the 3D structure of the environment, but also contains rich semantic information, which can provide strong support for fields such as autonomous driving and robot navigation.

[0092] Figure 2 for Figure 1 The specific execution flow diagram of step 102 of the embodiment shown in FIG. Figure 2 As shown, the following steps are included:

[0093] Step 201, acquiring two-dimensional image features based on time-series image data;

[0094] This step is to use the time-series image data of at least two viewing angles obtained in step 101 to obtain the corresponding two-dimensional image features. For example, for a multi-view time-series image sequence I = {I i , i=1,2,3,....,N}, extract the two-dimensional image features and obtain the two-dimensional image features of the above time series image sequence Wherein, N represents the number of image acquisition devices, and H, W, and C represent the height, width, and number of channels of the extracted two-dimensional image features, respectively.

[0095] Step 202: acquiring image depth estimation information of the two-dimensional image features based on device parameter information of the image acquisition device;

[0096] In this step, after acquiring the two-dimensional image features in step 201, the depth of the two-dimensional image features is estimated in combination with the device parameter information of the image acquisition device to obtain image depth estimation information of the two-dimensional image features. Where N represents the number of image acquisition devices, H, W, C depth They are the height, width and number of discrete depth values ​​contained in the extracted two-dimensional image feature points, respectively. The number of discrete depth values ​​usually refers to the number of different possible depth values ​​considered for each feature point in the depth estimation process. The device parameter information of the above-mentioned image acquisition device includes internal parameter information and external parameter information, wherein the internal parameter information includes information such as distortion coefficient, focal length, pixel size, etc.; the external parameter information includes information such as the position and rotation direction of the image acquisition device.

[0097] Step 203: Generate a three-dimensional bird's-eye view feature map based on the two-dimensional image features and the image depth estimation information of the two-dimensional image features.

[0098] This step is based on step 201 and step 202, based on the acquired two-dimensional image features and the image depth estimation information of the two-dimensional image features, and uses the perspective converter to generate a three-dimensional bird's-eye view feature map. Specifically, the two-dimensional image features and the depth estimation information of the two-dimensional image features are firstly outer-producted, and the result obtained after the outer-product operation is projected into the 3D space to obtain the three-dimensional image features. Where N represents the number of image acquisition devices, H, W, C channel , C depth After the height, width, number of channels, and number of discrete depth values ​​contained in the extracted two-dimensional image features are obtained, the voxel pooling operation is performed through the three-dimensional image features, and the three-dimensional image features under multiple perspectives are converted to a three-dimensional bird's-eye view feature map under a bird's-eye view, that is, a distributed grid constructed with the vehicle's own vehicle coordinates as the origin, and the three-dimensional cone features projected into the same grid are accumulated to generate a three-dimensional bird's-eye view feature map at the corresponding position. The above-mentioned perspective converter is used to convert the three-dimensional feature map of the image perspective into a three-dimensional bird's-eye view feature map.

[0099] In some embodiments, the first map information in the three-dimensional bird's-eye view feature map can be identified, and then a three-dimensional semantic map can be constructed according to the first map information, the three-dimensional point cloud frame and the vehicle's posture data. Specifically, Figure 3 Schematic diagram of a process of constructing a three-dimensional semantic map based on first map information, a three-dimensional point cloud frame and vehicle posture data in an embodiment of the present application, such as Figure 3 As shown, the following steps are included:

[0100] Step 301: Perform information recognition on the three-dimensional bird's-eye view feature map to obtain first map information in the three-dimensional bird's-eye view feature map;

[0101] In this step, after obtaining the three-dimensional bird's-eye view feature map in the above step 203, the corresponding information recognition algorithm is used to perform information recognition on the above three-dimensional bird's-eye view feature map to obtain the first map information in the three-dimensional bird's-eye view feature map. The above information recognition algorithm includes algorithms such as scene segmentation and target detection. The first map information includes static targets and dynamic targets in the time series image data, as well as attribute information and location information of the static targets and dynamic targets.

[0102] Step 302: construct a three-dimensional semantic map based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame, and the vehicle's posture data.

[0103] This step is to construct a three-dimensional semantic map using the first map information in the three-dimensional bird's-eye view feature map obtained in step 301, the vehicle posture data obtained in step 101, and the three-dimensional point cloud frame obtained in step 103.

[0104] In some embodiments, when information recognition is performed on the three-dimensional bird's-eye view feature map obtained in the above embodiment to obtain first map information in the three-dimensional bird's-eye view feature map, the first map information obtained includes at least one of the following two types of information:

[0105] One type is to perform scene semantic segmentation on the acquired three-dimensional bird's-eye view feature map to obtain the first map element information in the above three-dimensional bird's-eye view feature map, and the first map element information includes lane lines, curbs, traffic lights, and road sign light information.

[0106] The other type is to perform target detection on the acquired 3D bird's-eye view feature map to obtain the motion element information in the above 3D bird's-eye view feature map, and the motion element information includes the coordinate information, category, heading angle and other information of the detected target. For example, by using a 3D target detection algorithm to perform target detection on the 3D bird's-eye view feature map, moving targets such as motor vehicles, pedestrians, motorcycles, bicycles, traffic cones, etc. in the scene corresponding to the above 3D bird's-eye view feature map can be detected, and the 3D coordinate information, category, heading angle and other information of the above moving targets can be output.

[0107] In some embodiments, a three-dimensional semantic map can be constructed based on the first map information, three-dimensional point cloud frame and vehicle posture data in the three-dimensional bird's-eye view feature map. Specifically, the three-dimensional semantic map can be constructed using the first map element information, motion element information, three-dimensional point cloud frame and vehicle posture data in the three-dimensional bird's-eye view feature map. The three-dimensional semantic map constructed using the above information can achieve a comprehensive understanding of the vehicle's surrounding environment, provide accurate navigation and positioning services for autonomous driving vehicles, and support complex scene perception and decision-making braking, greatly improving the safety and reliability of the autonomous driving system.

[0108] In an embodiment of the present application, when obtaining the posture data of a vehicle on which an image acquisition device is installed, the posture data of the vehicle on which the image acquisition device is installed can be obtained based on time-series image data acquired by image acquisition devices of at least two perspectives and initial posture data acquired by at least one posture sensor.

[0109] In an embodiment of the present application, the above-mentioned posture sensor includes at least one of an inertial measurement unit, a global navigation system and a wheel speed meter, wherein the inertial measurement unit (IMU) is composed of three single-axis accelerometers and three single-axis gyroscopes, the accelerometer detects the acceleration signal of the object in the independent three-axis of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system, and after processing these signals, the posture of the object can be solved; the global satellite navigation system (GNSS) is an air-based radio navigation and positioning system that can provide users with all-weather three-dimensional coordinates, speed and time information at any location on the surface of the earth or in near-Earth space; the wheel speed meter is a sensor for measuring wheel speed, which is usually composed of a sensor and a signal processing circuit. The sensor is usually installed on the wheel to detect the rotation speed and direction of the tire, and the signal processing circuit is responsible for converting the signal output by the sensor into a digital signal that can be understood and used by the vehicle control system.

[0110] After acquiring the time-series image data and the vehicle posture data collected by the image acquisition device of at least two perspectives in the above embodiment, the above data is further processed to obtain the second map element information in the three-dimensional point cloud frame, specifically, Figure 4 As shown, the following steps are included:

[0111] Step 401: extracting feature points based on time-series image data collected by image collection devices of at least two viewing angles to obtain first feature point information;

[0112] This step is based on the time series image data collected by the image acquisition device of at least two perspectives, and the feature points are extracted from it using the corresponding feature point detection algorithm to obtain the first feature point information. The first feature point information is two-dimensional, and contains important information of the time series image data, such as position information, scale information, direction information, etc. Commonly used feature point detection algorithms include SIFT, SURF, ORB and other algorithms. In the process of extracting feature points from the time series image data using the above algorithms, not only the first feature point information can be obtained, but also the corresponding descriptor can be generated, which is a string of numbers obtained by encoding the local neighborhood of the feature points detected in the time series image data, which can uniquely identify and describe the above feature points for the subsequent feature point matching stage.

[0113] Step 402: obtaining a three-dimensional point cloud frame based on the first feature point information and the vehicle's posture data;

[0114] This step is based on step 401, and uses a triangulation algorithm to triangulate the obtained two-dimensional first feature point information and the corresponding vehicle posture data to obtain a three-dimensional point cloud frame. The above triangulation algorithm uses the principle of visual geometry to measure and analyze feature points or corresponding points under multiple perspectives to calculate the positions of these points in three-dimensional space. The three-dimensional point cloud frame is a collection of a group of discrete points in three-dimensional space. Based on the feature point information of multi-perspective time-series image data and the vehicle posture data, the obtained three-dimensional point cloud frame contains rich road semantic information and can better describe the changes in the ground.

[0115] Step 403: perform information recognition on the three-dimensional point cloud frame to obtain second map element information in the sparse three-dimensional point cloud frame.

[0116] This step is to obtain the second map element information in the 3D point cloud frame by performing information recognition through a corresponding information recognition algorithm, such as target recognition, target detection, etc., after obtaining the 3D point cloud frame in step 402. The second map element information includes lane lines, curbs, traffic lights, road signs, etc.

[0117] In some embodiments, after obtaining the three-dimensional point cloud frame, it is also necessary to use a global optimization function to optimize the point cloud coordinates of the above point cloud frame to obtain a sparse three-dimensional point cloud after optimization. The global optimization function includes the bundle adjustment method (BA), which is essentially an optimization model used to minimize the reprojection error, specifically, to minimize the difference between the projection of the real three-dimensional space point on the image plane and its secondary projection. In addition, BA can also be used to optimize the device parameters of the image acquisition device and determine the optimal device parameters, thereby reducing the cumulative error in subsequent calculations. For example, BA is used to optimize the point cloud coordinates in the three-dimensional point cloud frame to minimize the difference between the observation point and the projection point. The objective function can be set as:

[0118]

[0119] Among them is the matrix of observation points, u ij is the projection point, C j =(f j ,k ij ,k j ,R j ,t j ) is the equipment parameter of the jth image acquisition device, X i =(x i ,y i ,z i ) T is the three-dimensional coordinate of the i-th projection point, λ ij ={0,1} is a judgment parameter used to judge whether the i-th 3D point is visible in the j-th image acquisition device. If it is visible, its value is 1, otherwise it is 0. θ = (C1, C2, ..., C N ,X1,X2,…X M )∈R 3N*9M is the input device parameters of the image acquisition device and the set of three-dimensional point coordinates, M is the number of three-dimensional points, N is the number of image acquisition devices, e ij is the square of the projection error of the i-th 3D point on the j-th image acquisition device.

[0120] In an embodiment of the present application, information recognition of the three-dimensional point cloud frame includes performing information recognition on the optimized three-dimensional point cloud frame to obtain second map element information with higher accuracy and completeness.

[0121] In the above embodiment, after obtaining the optimized three-dimensional point cloud frame and the vehicle's posture data, the detection frame and two-dimensional detection results of the target in the three-dimensional point cloud frame can be extracted by clustering method, such as corner points, straight lines, contours, etc., and then based on the vehicle's posture data and after reprojecting the three-dimensional point cloud frame, it is matched with the two-dimensional detection result to verify whether the three-dimensional point cloud frame and the two-dimensional detection result are accurate representations of the same target.

[0122] In some embodiments, when constructing a three-dimensional semantic map, the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame, and the vehicle's posture data can be used. Specifically, this process can be based on the first map information in the three-dimensional bird's-eye view feature map, the second map element information in the three-dimensional point cloud frame, and the vehicle's posture data. Together, to construct a three-dimensional semantic map. This method integrates data from different sources, and the three-dimensional semantic map constructed with this data can achieve an accurate and comprehensive representation of the vehicle's surroundings. The process of constructing a three-dimensional semantic map using the above data is as follows: Figure 5 As shown, the following steps are included:

[0123] Step 501: performing fusion processing based on the first map information in the three-dimensional bird's-eye view feature map and the second map element information in the three-dimensional point cloud frame to obtain fused map element information and form a plurality of fused feature maps;

[0124] After obtaining the first map information in the three-dimensional bird's-eye view feature map and the second map element information in the three-dimensional point cloud frame in the above embodiment, the two types of data can be fused using a data fusion method, for example, direct fusion, feature-level fusion, decision-level fusion, model-based fusion, etc., to obtain fused map element information, and then extract the features of the fused map element information to form multiple fused feature maps.

[0125] Step 502: extract key frames from multiple fused feature maps;

[0126] This step is to determine whether the current fusion feature map is a key frame according to the degree of change of the vehicle's posture data relative to the previous key frame after obtaining multiple fusion feature maps in step 501, that is, extract key frames from the above multiple fusion feature maps, wherein the key frames are the most representative feature maps in the local similar fusion feature maps, which can be used to reduce information redundancy. For example, if the posture data of the vehicle changes greatly relative to the previous key frame when the fusion feature map is collected, the current fusion feature map is determined to be a key frame. And the degree of change of the vehicle's posture data relative to the previous key frame can be determined based on at least one of the information such as the rotation angle of the vehicle, the translation distance of the vehicle, the overlap rate or coverage rate of the field of view, the number and stability of feature points, and repeated observations. For example, when the rotation angle of the vehicle exceeds the rotation angle threshold, it is determined that the posture data of the vehicle has changed greatly relative to the previous key frame; when the current fusion feature map contains a sufficient number of stable feature points and the front and rear frames contain enough repeated observations, it is determined that the posture data of the vehicle has changed greatly relative to the previous key frame.

[0127] Step 503: construct a local three-dimensional semantic map based on the fused map element information included in the key frame and the vehicle's position data;

[0128] After the key frame is extracted in the above step 502, the fused map element information contained therein is used in combination with the vehicle's position data to construct a local three-dimensional semantic map.

[0129] Step 504: Based on the vehicle's position data, the local three-dimensional semantic maps are merged to obtain a global three-dimensional semantic map.

[0130] This step is to merge the local 3D semantic map based on the vehicle's position information and posture information contained in the vehicle's posture data to obtain a global 3D semantic map, which can solve the problems of poor stability and low query efficiency of the structural information of feature points represented by single-modal map elements when matching map elements, thereby improving the accuracy and efficiency of subsequent vehicle matching calculations during the positioning process.

[0131] In some embodiments, Figure 6 for Figure 5 The specific execution flow diagram of step 503 of the embodiment shown in FIG. Figure 6 As shown, the following steps are included:

[0132] Step 601: acquiring a vehicle coordinate system at an initial time based on the vehicle's position data, and using it as a map coordinate system;

[0133] This step uses the vehicle's longitude, latitude, altitude, roll angle, pitch angle, heading angle and other information in the vehicle's posture data to obtain the vehicle coordinate system at the initial moment, for example, the vehicle's rear axle coordinate system with the vehicle's rear axle center point as the origin. After that, the above vehicle coordinate system is used as the map coordinate system for the subsequent construction of a local three-dimensional semantic map.

[0134] Step 602: construct a local three-dimensional semantic map based on the map coordinate system and the fused map element information included in the key frame.

[0135] This step is based on step 601, and uses the fused map element information contained in the key frame to construct a local three-dimensional semantic map. Specifically, it is necessary to first convert the coordinate values ​​of the fused map elements from the image coordinate system to the map coordinate system, and then add the feature points in the key frame and the corresponding map element descriptors to the map to construct a local three-dimensional semantic map with semantic labels. The above-mentioned image coordinate system is a coordinate system with the center of the image plane as the origin and two vertical edges parallel to the image plane as the x-axis and y-axis respectively; the feature points in the key frame can be stably detected under different viewing angles and lighting conditions, and are distributed as evenly as possible in the image. In addition, these feature points have consistency in spatial or geometric properties across viewing angles; map elements include lane lines, curbs, traffic lights, road signs, etc.; each element in the map element, such as a closed polygon, a node, a polyline, a line segment, etc., is in the map coordinate system.

[0136] In the above embodiment, when the coordinate values ​​of the fused map elements are converted from the image coordinate system to the map coordinate system, the following conversion relationship may be used to complete the conversion of the above coordinate values:

[0137]

[0138] The X v , Y v , Z v is the 3D coordinate of the fused map element in the i-th key frame in the map coordinate system, u i 、v i are the pixel coordinates of the ith image acquisition device relative to the temporal image plane, are the equipment parameters of the i-th image acquisition device, The coordinate value of the fused map element in the i-th frame is converted from the image coordinate system corresponding to the image acquisition device to the posture and position offset of the map coordinate system. The above posture can be represented by quaternion. Z is the depth value of the i-th fused map element.

[0139] In the embodiment of the present application, after constructing a local three-dimensional semantic map based on the map coordinate system and the fused map element information included in the key frame, it is also necessary to convert the coordinate values ​​of the fused map element information in the local three-dimensional semantic map into coordinate values ​​in the global coordinate system. The global coordinate system is the world coordinate system, which is a fixed coordinate system relative to the surface of the earth. In this coordinate system, the positions of objects such as vehicles and map elements are measured relative to a fixed point on the surface of the earth. The above conversion process can be completed using the following conversion relationship:

[0140] [X w Y w Z w ] T =R(q i )[X v Y v Z v ] T +p i

[0141] The X w , Y w , Z w To fuse the three-dimensional coordinates of map elements in the global coordinate system, X v , Y v , Z v To fuse the three-dimensional coordinates of map elements in the map coordinate system, R(q i ), p i The coordinate values ​​of the fused map elements of the i-th key frame are converted from the map coordinate system to the posture and position offset in the global coordinate system.

[0142] In the embodiment of the present application, after constructing a plurality of local 3D semantic maps, the local 3D semantic maps can be merged according to the vehicle's posture data to obtain a global 3D semantic map. The specific process is as follows: Figure 7 As shown, the following steps are included:

[0143] Step 701: Generate a vehicle motion trajectory based on the vehicle's position and posture data;

[0144] This step uses the posture data provided by sensors such as the inertial measurement unit, global satellite navigation system, and wheel speed meter to interpolate and compensate for the changes in posture data between key frames, and then generate the vehicle's motion trajectory based on the posture data of the above vehicle. Specifically, the vehicle's rotation angle, translation distance and other data are interpolated based on the data provided by the above sensors, and the posture data is corrected to reduce the cumulative error, and then a smooth and continuous vehicle motion trajectory is generated based on the above optimized posture data. The above inertial measurement unit provides the vehicle's acceleration and angular velocity information; the global satellite navigation system provides the vehicle's positioning information in an outdoor environment; and the wheel speed meter directly provides the vehicle's speed and direction information.

[0145] Step 702: merge the local three-dimensional semantic maps based on the vehicle motion trajectory to obtain a global three-dimensional semantic map.

[0146] In this step, after the vehicle motion trajectory is generated in step 701, multiple local 3D semantic maps are merged according to the order of vehicle travel based on the vehicle motion trajectory to obtain a global 3D semantic map. By merging multiple local 3D semantic maps based on the vehicle motion trajectory to obtain a global 3D semantic map, it can be ensured that the global 3D semantic map can accurately reflect the environmental features on the vehicle's driving path, such as the shape of the road, the location of obstacles, and the layout of surrounding buildings.

[0147] In an embodiment of the present application, a tree data structure can be used to store a global three-dimensional semantic map, and the fused map element information in multiple local three-dimensional semantic maps is stored in the nodes of the above tree data structure. The tree data structure is composed of a root node and several child nodes, and these child nodes can have their own child nodes, thereby forming a recursive structure, which provides an efficient way to store and access data through its hierarchical organization, making various complex operations feasible and efficient. Commonly used tree data structures include binary trees, balanced trees, B trees, and octrees. For example, a global three-dimensional semantic map constructed by octree storage, the nodes of the octree store the fused map element information in multiple local three-dimensional semantic maps, specifically, the fused map element information in multiple local three-dimensional semantic maps is first collected by sampling at equal intervals, and then the collected fused map element information is stored in the nodes of the octree storage structure of a preset resolution, and each node stores the semantic label of the fused map element, the three-dimensional coordinates, the probability between adjacent nodes, the association relationship between map elements and other information. By storing the constructed global three-dimensional semantic map in a tree-like data structure, the above map can be used in vehicle positioning or navigation applications, allowing the autonomous driving system to quickly query, match and accurately reflect the relative inertia of the vehicle and its surrounding targets in space. In addition, the query or matching results can also be directly used in the analysis and decision-making part of subsequent regulatory control modules, thereby improving the driving safety of vehicles in dense urban scenes and complex road conditions.

[0148] Each node of the octree in the above embodiment represents a volume element of a cube, and each node has eight child nodes. The volume elements represented by the eight child nodes added together equal the volume of the parent node. The octree is used to store global three-dimensional semantic map information and local three-dimensional semantic map information, and the above map information can be gradually refined in three-dimensional space, and can cope with the precise positioning and adjustment of local areas during subsequent map increments and updates. It can also effectively cope with the changes and adjustments of the status of new elements or existing elements in the map through the dynamic merging and splitting of its nodes. In addition, the deletion or insertion of nodes can be realized simply and efficiently in this structure.

[0149] In the above embodiment, the process of merging the local three-dimensional semantic map based on the generated vehicle motion trajectory is as follows: Figure 8 As shown, the following steps are included:

[0150] Step 801: determine whether the key frame for constructing the local three-dimensional semantic map includes a loop candidate frame;

[0151] The loop candidate frames in this step refer to those key frames that may trigger loop detection. Specifically, when a new key frame is added to the map, it will be selected from the existing key frames through certain screening and matching strategies. The key frames that may form a loop with the current new key frame will be further determined through steps such as position recognition and geometric verification to determine whether a loop is actually formed. The above loop detection refers to the process of identifying the vehicle's return to a previously visited location during the vehicle's positioning and map construction process. It can eliminate the accumulated errors in the process of building a three-dimensional semantic map to ensure the consistency of the vehicle's motion trajectory and the three-dimensional semantic map.

[0152] In an embodiment of the present application, in order to determine whether a key frame for constructing a local three-dimensional semantic map contains a loop candidate frame, a variety of methods are used. Specifically, it can be achieved by at least one of global coordinate matching, bag-of-words model or feature database matching. The above-mentioned global coordinate matching method is to determine whether a new key frame is a loop key frame by comparing the global coordinates of the new key frame with the existing key frame. For example, if the global coordinates of the above two key frames are similar, they may be captured at the same location, thereby determining that the new key frame is a loop candidate frame.

[0153] The bag-of-words model-based method converts the pre-built dictionary into a vector description, and then compares the visual similarity between the new keyframe and the existing keyframe to determine whether the new keyframe is a loop keyframe. For example, based on the bag-of-words model BOW, the visual similarity between the new keyframe and the existing keyframe is compared. If the visual similarity between the two keyframes is high, the new keyframe is determined to be a loop candidate frame.

[0154] The feature database matching method is based on a pre-constructed feature database, which has stored the feature information of existing key frames. After that, the feature descriptors extracted from the feature points of the new key frame are matched with the descriptors of the existing key frames in the feature database, and the new key frame is judged to be a loop key frame based on the matching results, the number of matching pairs and the geometric relationship of the matching pairs. For example, if the new key frame can match enough existing key frames, and these matching pairs present a certain geometric relationship in space, such as colinearity, coplanarity, etc., then the new key frame is judged to be a loop candidate frame. Step 802: When the local three-dimensional semantic map includes a loop candidate frame, the pose of the local three-dimensional semantic map is globally optimized based on the constructed global three-dimensional semantic map;

[0155] After determining in step 801 that the key frame for constructing the local 3D semantic map includes the loop candidate frame, the BA optimization function is used, and based on the constructed global 3D semantic map, the pose of the local 3D semantic map is globally optimized. This process can ensure the global consistency between the local 3D semantic map and the global 3D semantic map, thereby laying the foundation for the subsequent construction of a new high-precision global 3D semantic map. The BA optimization function can be set to:

[0156]

[0157] where [r0,t0,r1,t1,…,r k ,t k ] is the pose of the local 3D semantic map, is the relative position of the kth and k+1th key frames of the local 3D semantic map obtained based on the motion trajectory, is the relative pose at the i-th and j-th moments in the corresponding loop candidate frame, function f(·) is the function for calculating the relative pose between local 3D semantic maps, and ζ is the set of loop candidate frames.

[0158] Step 803: Merge the local three-dimensional semantic maps after global optimization based on the vehicle motion trajectory.

[0159] This step is based on step 802, and merges the local three-dimensional semantic maps after global optimization according to the vehicle motion trajectory to obtain a new global three-dimensional semantic map.

[0160] In an embodiment of the present application, after the global three-dimensional semantic map is constructed, it is also necessary to update the above map based on the newly added time-series image data, sensor data, scene data, etc. Specifically, it is necessary to spatially align and fuse the newly added data with the existing global three-dimensional semantic map to ensure the spatiotemporal consistency of the map. At the same time, it is also necessary to regularly update the tree data structure, including merging map elements at the same position, updating their semantic labels, etc., and performing global consistency detection on it to identify and repair abnormal map areas caused by sensor noise, dynamic target interference, etc., and for areas that have not been observed for a long time, the confidence of their semantic information can be appropriately reduced according to historical information and uncertainty models. In addition, in order to make the above-mentioned global three-dimensional semantic map meet the actual application needs, it is also necessary to compress, refine and other operations to ensure the effectiveness and efficiency of the global three-dimensional semantic map. Updating the constructed global three-dimensional semantic map based on the tree data structure can improve the maintenance efficiency and accuracy of the global three-dimensional semantic map, thereby providing powerful and flexible map support for the vehicle's autonomous driving, thereby ensuring that the vehicle can continue to accurately locate and navigate in a complex and changing real environment.

[0161] Fig. 9 FIG. 1 is a flow chart of a vehicle positioning method in an embodiment of the present application, such as Fig. 9 As shown, the following steps are included:

[0162] Step 901: Based on Figure 1-Figure 8 The method for constructing a three-dimensional semantic map recorded in any one of the embodiments, constructing a three-dimensional semantic map;

[0163] Step 902: Position the vehicle based on the three-dimensional semantic map.

[0164] This step is performed after successfully constructing a three-dimensional semantic map in step 901, for example, a local three-dimensional semantic map, a global three-dimensional semantic map, etc., and positioning the vehicle based on the three-dimensional semantic map.

[0165] As shown above, by constructing a three-dimensional semantic map using the three-dimensional semantic map construction method in the above embodiment, and positioning the vehicle based on the constructed three-dimensional semantic map, the vehicle can not only rely on geometric features but also combine semantic information to determine its own position, thereby improving the accuracy and reliability of vehicle positioning, and further improving the safety of the autonomous driving system.

[0166] Fig.10 FIG. 1 is a flow chart of a vehicle navigation method in an embodiment of the present application. Fig.10 As shown, the following steps are included:

[0167] Step 1001: Based on Figure 1-Figure 8 The method for constructing a three-dimensional semantic map recorded in any one of the embodiments, constructing a three-dimensional semantic map;

[0168] Step 1002: Perform vehicle navigation based on the three-dimensional semantic map.

[0169] The step is that after successfully constructing a three-dimensional semantic map in step 1001, for example, a local three-dimensional semantic map, a global three-dimensional semantic map, etc., the vehicle is navigated according to the three-dimensional semantic map.

[0170] As described above, the three-dimensional semantic map constructed by the three-dimensional semantic map construction method in the above embodiment not only depicts the geometric structure of the physical space, but also incorporates various types of semantic information. When the above-mentioned three-dimensional semantic map is used to navigate the vehicle, the vehicle can make full use of the semantic information in the map to make more intelligent decisions, thereby improving the accuracy and safety of navigation.

[0171] With the above Figure 1-Figure 8 Corresponding to the provided method for constructing a three-dimensional semantic map, the embodiments of the present application provide a corresponding device. Fig.11This is a schematic diagram of a structure of a device for constructing a three-dimensional semantic map in an example of this application, such as Fig.11 As shown, the construction device of the three-dimensional semantic map includes a data acquisition module 11, a feature map acquisition module 12, a three-dimensional point cloud frame acquisition module 13, and a map construction module 14, wherein the data acquisition module 11 is used to acquire time-series image data acquired by image acquisition devices of at least two perspectives, and acquire the posture data of the vehicle on which the image acquisition device is installed; the feature map acquisition module 12 is used to acquire a three-dimensional bird's-eye view feature map based on the time-series image data acquired by image acquisition devices of at least two perspectives; the three-dimensional point cloud frame acquisition module 13 is used to obtain a sparse three-dimensional point cloud frame based on the time-series image data acquired by image acquisition devices of at least two perspectives and the posture data of the vehicle; the map construction module 14 is used to construct a three-dimensional semantic map based on the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame.

[0172] As described above, the three-dimensional semantic map constructed by the above-mentioned three-dimensional semantic map construction device is based on time-series image data and fully considers the failure of image features in weak textures and medium and high-speed scenes. Using it for vehicle positioning and navigation can improve the map matching efficiency and positioning robustness of the vehicle during the positioning and navigation process, thereby improving the safety of the autonomous driving system.

[0173] Acquire time-series image data acquired by image acquisition devices of at least two viewing angles, and acquire position and posture data of a vehicle on which the image acquisition devices are installed;

[0174] Obtaining a three-dimensional bird's-eye view feature map based on time-series image data collected by image acquisition devices of at least two viewing angles;

[0175] Obtaining a three-dimensional point cloud frame based on time-series image data collected by image acquisition devices of at least two viewing angles and position data of the vehicle;

[0176] A 3D semantic map is constructed based on the 3D bird's-eye view feature map and the 3D point cloud frame.

[0177] In some embodiments, the three-dimensional semantic map is constructed based on the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame, including:

[0178] A 3D semantic map is constructed based on the 3D bird's-eye view feature map, 3D point cloud frame and vehicle posture data.

[0179] In some embodiments, the three-dimensional bird's-eye view feature map is obtained by using the time-series image data collected by the image collection device based on at least two viewing angles, including:

[0180] Acquiring two-dimensional image features based on time-series image data;

[0181] Acquire image depth estimation information of two-dimensional image features based on device parameter information of the image acquisition device;

[0182] A three-dimensional bird's-eye view feature map is generated based on the two-dimensional image features and image depth estimation information of the two-dimensional image features.

[0183] In some embodiments, the above-mentioned three-dimensional bird's-eye view feature map, three-dimensional point cloud frame and vehicle posture data include:

[0184] Performing information recognition on the three-dimensional bird's-eye view feature map to obtain first map information in the three-dimensional bird's-eye view feature map;

[0185] A three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame, and the vehicle's posture data.

[0186] In some embodiments, the above-mentioned information recognition of the three-dimensional bird's-eye view feature map to obtain the first map information in the three-dimensional bird's-eye view feature map includes at least one of the following:

[0187] Performing scene semantic segmentation on the three-dimensional bird's-eye view feature map to obtain first map element information in the three-dimensional bird's-eye view feature map;

[0188] Target detection is performed on the 3D bird's-eye view feature map to obtain the motion element information in the 3D bird's-eye view feature map.

[0189] In some embodiments, the three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame, and the vehicle's position data, including:

[0190] A three-dimensional semantic map is constructed based on the first map element information, motion element information and vehicle posture data in the three-dimensional bird's-eye view feature map.

[0191] In some embodiments, the three-dimensional point cloud frame is obtained by using the time-series image data and the vehicle posture data collected by the image acquisition device based on at least two viewing angles, including:

[0192] Extracting feature points based on time-series image data collected by image collection devices of at least two viewing angles to obtain first feature point information;

[0193] Obtaining a three-dimensional point cloud frame based on the first feature point information and the position and posture data of the vehicle;

[0194] Information recognition is performed on the three-dimensional point cloud frame to obtain second map element information in the three-dimensional point cloud frame.

[0195] In some embodiments, after obtaining the three-dimensional point cloud frame, the method further includes:

[0196] Optimizing the point cloud coordinates in the three-dimensional point cloud frame based on the global optimization function to obtain an optimized three-dimensional point cloud frame;

[0197] Performing information recognition on the three-dimensional point cloud frame includes:

[0198] Information recognition is performed on the optimized three-dimensional point cloud frame to obtain second map element information in the three-dimensional point cloud frame.

[0199] In some embodiments, the three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame, and the vehicle's position data, including:

[0200] A three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the second map element information in the three-dimensional point cloud frame, and the vehicle's posture data.

[0201] In some embodiments, the three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the second map element information in the three-dimensional point cloud frame, and the vehicle's position data, including:

[0202] Based on the first map information in the three-dimensional bird's-eye view feature map and the second map element information in the three-dimensional point cloud frame, a fusion process is performed to obtain fused map element information and form a plurality of fused feature maps;

[0203] Extract key frames from multiple fused feature maps;

[0204] Based on the fused map element information included in the key frames and the vehicle's position data, a local 3D semantic map is constructed;

[0205] Based on the vehicle's posture data, the local 3D semantic maps are merged to obtain a global 3D semantic map.

[0206] In some embodiments, the above-mentioned construction of a local three-dimensional semantic map based on the fused map element information included in the key frame and the vehicle's position data includes:

[0207] The vehicle coordinate system at the initial moment is obtained based on the vehicle's posture data and used as the map coordinate system;

[0208] Based on the map coordinate system and the fused map element information included in the key frames, a local three-dimensional semantic map is constructed.

[0209] In some embodiments, after constructing the local three-dimensional semantic map based on the map coordinate system and the fused map element information included in the key frame, the following further includes:

[0210] The coordinate values ​​of the fused map element information in the local 3D semantic map are converted into coordinate values ​​in the global coordinate system.

[0211] In some embodiments, the above-mentioned merging process of the local 3D semantic map based on the vehicle's position data to obtain the global 3D semantic map includes:

[0212] Generate vehicle motion trajectory based on vehicle posture data;

[0213] The local 3D semantic maps are merged based on the vehicle motion trajectory to obtain a global 3D semantic map.

[0214] In some embodiments, the global three-dimensional semantic map is stored in a tree data structure, and the fused map element information of the local three-dimensional semantic map is stored in the nodes of the tree data structure.

[0215] In some embodiments, the above-mentioned merging process of the local three-dimensional semantic map based on the vehicle motion trajectory includes:

[0216] Determine whether the key frame for constructing the local three-dimensional semantic map includes a loop candidate frame;

[0217] When the local 3D semantic map includes loop candidate frames, the pose of the local 3D semantic map is globally optimized based on the constructed global 3D semantic map;

[0218] The local three-dimensional semantic maps after global optimization are merged based on the vehicle motion trajectory.

[0219] In some embodiments, the above-mentioned determination of whether the key frame for constructing the local three-dimensional semantic map includes a loop candidate frame includes:

[0220] Whether a key frame for constructing a local three-dimensional semantic map includes a loop candidate frame is determined by at least one of global coordinate matching, bag-of-words model, or feature database matching.

[0221] With the above Fig. 9 Corresponding to the provided vehicle positioning method, the embodiments of the present application provide a corresponding device. Fig.12 This is a schematic diagram of the structure of a vehicle positioning device in an example of this application, such as Fig.12As shown, the vehicle positioning device includes a data acquisition module 11, a feature map acquisition module 12, a three-dimensional point cloud frame acquisition module 13, a map construction module 14, and a vehicle positioning module 21, wherein the data acquisition module 11 is used to acquire time-series image data acquired by image acquisition devices of at least two viewing angles, and acquire the posture data of the vehicle on which the image acquisition device is installed; the feature map acquisition module 12 is used to acquire a three-dimensional bird's-eye view feature map based on the time-series image data acquired by image acquisition devices of at least two viewing angles; the three-dimensional point cloud frame acquisition module 13 is used to obtain a three-dimensional point cloud frame based on the time-series image data acquired by image acquisition devices of at least two viewing angles and the posture data of the vehicle; the map construction module 14 is used to construct a three-dimensional semantic map based on the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame; the vehicle positioning module 21 is used to perform vehicle positioning based on the three-dimensional semantic map constructed by the three-dimensional semantic map construction device.

[0222] As described above, by using the above-mentioned vehicle positioning device to locate the vehicle, the vehicle can not only rely on geometric features but also combine semantic information to determine its own position, thereby improving the accuracy and reliability of vehicle positioning, and further improving the safety of the autonomous driving system.

[0223] With the above Fig.10 Corresponding to the provided vehicle navigation method, the embodiments of the present application provide a corresponding device. Fig.13 This is a schematic diagram of the structure of a vehicle navigation device in an example of this application, such as Fig.13 As shown, the vehicle navigation device includes a data acquisition module 11, a feature map acquisition module 12, a three-dimensional point cloud frame acquisition module 13, a map construction module 14, and a vehicle navigation module 31, wherein the data acquisition module 11 is used to acquire time-series image data acquired by image acquisition devices of at least two viewing angles, and acquire the posture data of a vehicle on which the image acquisition device is installed; the feature map acquisition module 12 is used to acquire a three-dimensional bird's-eye view feature map based on the time-series image data acquired by image acquisition devices of at least two viewing angles; the three-dimensional point cloud frame acquisition module 13 is used to obtain a three-dimensional point cloud frame based on the time-series image data acquired by image acquisition devices of at least two viewing angles and the posture data of the vehicle; the map construction module 14 is used to construct a three-dimensional semantic map based on the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame; the vehicle navigation module 31 is used to perform vehicle navigation based on the three-dimensional semantic map constructed by the three-dimensional semantic map construction device.

[0224] As described above, by navigating the vehicle through the above-mentioned vehicle navigation device, the vehicle can make full use of the semantic information in the map to make more intelligent decisions, thereby improving the accuracy and safety of navigation.

[0225] The present application also provides an electronic device, such as Fig.14As shown, the electronic device includes: a processor 41 and a memory 42, the memory 42 stores programs or instructions that can be run on the processor, and the above-mentioned programs or instructions are executed by the processor to achieve the following Figure 1-Figure 8 The steps of the method for constructing a three-dimensional semantic map; or, when the program or instruction is executed by the processor, the following is achieved: Fig. 9 The steps of the vehicle positioning method; or, when the program or instruction is executed by the processor, the following is achieved: Fig.10 The steps of the vehicle navigation method.

[0226] The present application also provides a vehicle, such as Fig.15 As shown, the vehicle includes Fig.14 The electronic device 51, wherein the memory of the electronic device stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the following is achieved: Figure 1-Figure 8 The steps of the method for constructing a three-dimensional semantic map; or, when the program or instruction is executed by the processor, the following is achieved: Fig. 9 The steps of the vehicle positioning method; or, when the program or instruction is executed by the processor, the following is achieved: Fig.10 The steps of the vehicle navigation method.

[0227] The present application also provides a computer-readable storage medium, which implements the following when the above program or instruction is executed by the processor: Figure 1-Figure 8 The steps of the method for constructing a three-dimensional semantic map; or, when the program or instruction is executed by the processor, the following is achieved: Fig. 9 The steps of the vehicle positioning method; or, when the program or instruction is executed by the processor, the following is achieved: Fig.10 The steps of the vehicle navigation method.

[0228] The present application also provides a computer program product, which implements the following when the above program or instruction is executed by the processor: Figure 1-Figure 8 The steps of the method for constructing a three-dimensional semantic map; or, when the program or instruction is executed by the processor, the following is achieved: Fig. 9 The steps of the vehicle positioning method; or, when the program or instruction is executed by the processor, the following is achieved: Fig.10 The steps of the vehicle navigation method.

[0229] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0230] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0231] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A method for constructing a three-dimensional semantic map, characterized in that: include: Acquire time-series image data acquired by image acquisition devices of at least two viewing angles, and acquire position and posture data of a vehicle on which the image acquisition device is installed; Obtaining a three-dimensional bird's-eye view feature map based on the time-series image data collected by the image collection device of the at least two viewing angles; Obtaining a three-dimensional point cloud frame based on the time-series image data collected by the image acquisition device of the at least two viewing angles and the posture data of the vehicle; A three-dimensional semantic map is constructed based on the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame.

2. The method according to claim 1, characterized in that The constructing of a three-dimensional semantic map based on the three-dimensional bird's-eye view feature map and the three-dimensional point cloud frame includes: A three-dimensional semantic map is constructed based on the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame and the position data of the vehicle.

3. The method according to claim 2, characterized in that The three-dimensional bird's-eye view feature map is obtained based on the time-series image data collected by the image collection device of the at least two viewing angles, including: Acquire two-dimensional image features based on the time-series image data; Acquire image depth estimation information of the two-dimensional image feature based on device parameter information of the image acquisition device; A three-dimensional bird's-eye view feature map is generated based on the two-dimensional image features and the image depth estimation information of the two-dimensional image features.

4. The method according to claim 2, characterized in that: The constructing of a three-dimensional semantic map based on the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame and the position data of the vehicle includes: Performing information recognition on the three-dimensional bird's-eye view feature map to obtain first map information in the three-dimensional bird's-eye view feature map; A three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame and the posture data of the vehicle.

5. The method according to claim 4, characterized in that The performing information recognition on the three-dimensional bird's-eye view feature map to obtain the first map information in the three-dimensional bird's-eye view feature map includes at least one of the following: Performing scene semantic segmentation on the three-dimensional bird's-eye view feature map to obtain first map element information in the three-dimensional bird's-eye view feature map; Perform target detection on the three-dimensional bird's-eye view feature image to obtain motion element information in the three-dimensional bird's-eye view feature image.

6. The method according to claim 5, characterized in that The constructing of a three-dimensional semantic map based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame and the posture data of the vehicle includes: A three-dimensional semantic map is constructed based on the first map element information, the motion element information, the three-dimensional point cloud frame and the position and posture data of the vehicle in the three-dimensional bird's-eye view feature map.

7. The method according to any one of claims 4 to 6, characterized in that: The method of acquiring a three-dimensional point cloud frame based on the time-series image data acquired by the image acquisition device of the at least two viewing angles and the posture data of the vehicle comprises: Extracting feature points based on the time-series image data collected by the image collection device of the at least two viewing angles to obtain first feature point information; Obtaining a three-dimensional point cloud frame based on the first feature point information and the posture data of the vehicle; Information recognition is performed on the three-dimensional point cloud frame to obtain second map element information in the three-dimensional point cloud frame.

8. The method according to claim 7, characterized in that After obtaining the three-dimensional point cloud frame, the method further includes: Optimizing the point cloud coordinates in the three-dimensional point cloud frame based on a global optimization function to obtain an optimized three-dimensional point cloud frame; The performing information recognition on the three-dimensional point cloud frame includes: Information recognition is performed on the optimized three-dimensional point cloud frame to obtain second map element information in the three-dimensional point cloud frame.

9. The method according to claim 8, characterized in that The constructing of a three-dimensional semantic map based on the first map information in the three-dimensional bird's-eye view feature map, the three-dimensional point cloud frame and the posture data of the vehicle includes: A three-dimensional semantic map is constructed based on the first map information in the three-dimensional bird's-eye view feature map, the second map element information in the three-dimensional point cloud frame, and the position data of the vehicle.

10. The method according to claim 9, characterized in that The constructing of a three-dimensional semantic map based on the first map information in the three-dimensional bird's-eye view feature map, the second map element information in the three-dimensional point cloud frame and the position data of the vehicle includes: Performing fusion processing based on the first map information in the three-dimensional bird's-eye view feature map and the second map element information in the three-dimensional point cloud frame to obtain fused map element information and form a plurality of fused feature maps; Extracting key frames from the multiple fused feature maps; Constructing a local three-dimensional semantic map based on the fused map element information included in the key frame and the position data of the vehicle; Based on the position data of the vehicle, the local three-dimensional semantic maps are merged to obtain a global three-dimensional semantic map.

11. The method according to claim 10, characterized in that The step of constructing a local three-dimensional semantic map based on the fused map element information included in the key frame and the position data of the vehicle includes: Acquire a vehicle coordinate system at an initial moment based on the position and posture data of the vehicle and use it as a map coordinate system; A local three-dimensional semantic map is constructed based on the map coordinate system and the fused map element information included in the key frame.

12. The method according to claim 11, characterized in that After constructing the local three-dimensional semantic map based on the map coordinate system and the fused map element information included in the key frame, the method further includes: The coordinate values ​​of the fused map element information in the local three-dimensional semantic map are converted into coordinate values ​​in the global coordinate system.

13. The method according to claim 12, characterized in that Based on the position data of the vehicle, the local three-dimensional semantic map is merged to obtain a global three-dimensional semantic map, including: Generating a vehicle motion trajectory based on the position and posture data of the vehicle; Based on the vehicle motion trajectory, the local three-dimensional semantic maps are merged to obtain a global three-dimensional semantic map.

14. The method according to claim 13, characterized in that The global three-dimensional semantic map is stored in a tree data structure, and the fused map element information of the local three-dimensional semantic map is stored in the nodes of the tree data structure.

15. The method according to claim 13, characterized in that The merging process of the local three-dimensional semantic map based on the vehicle motion trajectory includes: Determining whether a key frame for constructing the local three-dimensional semantic map includes a loop candidate frame; When the local three-dimensional semantic map includes loop candidate frames, globally optimizing the position and posture of the local three-dimensional semantic map based on the constructed global three-dimensional semantic map; The local three-dimensional semantic map after global optimization is merged based on the vehicle motion trajectory.

16. The method according to claim 15, characterized in that The determining whether the key frame for constructing the local three-dimensional semantic map includes a loop candidate frame includes: Whether the key frame for constructing the local three-dimensional semantic map includes a loop candidate frame is determined by at least one of global coordinate matching, bag-of-words model or feature database matching.

17. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method for constructing a three-dimensional semantic map as described in any one of claims 1 to 16 are implemented.

18. A vehicle, characterized in that: An electronic device comprising the electronic device of claim 17, wherein the memory of the electronic device stores a program or instruction that can be executed on a processor, and the program or instruction, when executed by the processor, implements the steps of the method for constructing a three-dimensional semantic map as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for constructing a three-dimensional semantic map as described in any one of claims 1 to 16 are implemented.

20. A computer program product, characterized in that When the program or instruction is executed by a processor, the steps of the method for constructing a three-dimensional semantic map as described in any one of claims 1 to 16 are implemented.

Images