Semantic map construction method, device, electronic device, and storage medium

Abstract

The present application discloses a semantic map construction method, device, electronic device, and storage medium. The method includes: obtaining a first keyframe image and its corresponding road semantic segmentation results and corresponding high-precision positioning data; determining road key points in the first keyframe image and determining corresponding road three-dimensional points based on their corresponding high-precision positioning data; determining road element three-dimensional points corresponding to the first keyframe image using a preset oblique projection algorithm based on the road semantic segmentation results and the road three-dimensional points; constructing a semantic map based on the road element three-dimensional points to obtain a constructed semantic map. The present application utilizes road semantic segmentation results and high-precision positioning data to construct a semantic map based on road elements. This is not affected by any degree of vehicle body fluctuation, and does not require pre-calibration of camera-to-ground external parameters. It is more applicable to different scenarios and has high mapping accuracy, providing a reliable positioning basis for autonomous driving scenarios.

Description

Technical Field

[0001] The present application relates to the field of map construction technology, and in particular to a semantic map construction method, device, electronic device, and storage medium. Background Art

[0002] In the field of autonomous driving, high-precision maps are a crucial component for achieving high-level autonomous driving. Semantic maps built using visual technology offer advantages such as low production costs, minimal memory usage, and accurate semantic information. They have been widely used for real-time positioning in scenarios such as underground parking lots and urban roads. However, existing semantic map construction solutions still face the following challenges:

[0003] 1) The IPM (Inverse Perspective Mapping) algorithm commonly used in existing semantic map construction solutions assumes a flat ground surface, resulting in large mapping errors for road scenes with significant undulations.

[0004] 2) Some existing improved or adaptive IPM algorithms can only correct the pitch and roll angles between the camera and the ground to a certain extent, but cannot accurately estimate the height between the camera and the ground in real time. Therefore, mapping errors caused by the undulation of the vehicle body still exist. Summary of the Invention

[0005] The embodiments of the present application provide a semantic map construction method, device, electronic device, and storage medium to improve the mapping accuracy of semantic maps and provide accurate positioning basis for autonomous driving scenarios.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for constructing a semantic map, wherein the method includes:

[0008] Acquire a first key frame image, and a road semantic segmentation result and corresponding high-precision positioning data corresponding to the first key frame image;

[0009] Determining a road surface key point in the first key frame image, and determining a road surface three-dimensional point corresponding to the road surface key point in the first key frame image based on high-precision positioning data corresponding to the first key frame image;

[0010] Determining the 3D points of road elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road semantic segmentation result corresponding to the first key frame image and the 3D points of the road surface;

[0011] A semantic map is constructed according to the three-dimensional points of the road surface elements to obtain a constructed semantic map.

[0012] Optionally, determining road surface key points in the first key frame image includes:

[0013] performing histogram equalization processing on the first key frame image to obtain a processed first key frame image;

[0014] Determining key points in the processed first key frame image using a preset tracking algorithm;

[0015] The key points in the processed first key frame image are filtered using a preset filtering strategy to obtain filtered key points in the first key frame image as road surface key points in the first key frame image.

[0016] Optionally, determining the road surface three-dimensional point corresponding to the road surface key point in the first key frame image according to the high-precision positioning data corresponding to the first key frame image includes:

[0017] Determining a camera pose corresponding to the first key frame image based on the high-precision positioning data and external parameters from the camera to the vehicle body;

[0018] Based on the camera pose corresponding to the first key frame image, a preset triangulation algorithm is used to determine a road surface three-dimensional point corresponding to a road surface key point in the first key frame image.

[0019] Optionally, determining the 3D points of road surface elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road surface semantic segmentation result corresponding to the first key frame image and the 3D points of the road surface includes:

[0020] Performing plane fitting on the three-dimensional points of the road surface corresponding to the first key frame image to obtain a plane fitting result corresponding to the first key frame image;

[0021] According to the road surface semantic segmentation result corresponding to the first key frame image and the plane fitting result, a preset oblique projection algorithm is used to determine the three-dimensional points of the road surface elements corresponding to the first key frame image.

[0022] Optionally, the plane fitting result includes a plane fitting equation, the road surface semantic segmentation result includes a road surface element segmentation image, and determining the three-dimensional points of the road surface elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road surface semantic segmentation result corresponding to the first key frame image and the plane fitting result includes:

[0023] Extracting contour points from the road surface element segmentation image to obtain road surface element contour points;

[0024] Converting the pavement element contour points to a camera normalized plane and performing unit vectorization processing to obtain unit vectors corresponding to the pavement element contour points;

[0025] Determining a spatial straight line equation of the unit vector corresponding to the pavement element contour point in a camera coordinate system based on the unit vector corresponding to the pavement element contour point and the camera optical center;

[0026] The three-dimensional point of the road surface element corresponding to the first key frame image is determined according to the intersection of the spatial straight line equation of the unit vector corresponding to the road surface element contour point in the camera coordinate system and the plane fitting equation.

[0027] Optionally, the first key frame image is located in a preset sliding window. After constructing a semantic map based on the three-dimensional points of the road surface elements to obtain the constructed semantic map, the method further includes:

[0028] Determining whether the three-dimensional points of road surface elements corresponding to each key frame image in the preset sliding window meet a preset deletion condition;

[0029] If satisfied, the road surface element three-dimensional point is deleted from the preset sliding window.

[0030] Optionally, the first key frame image is located in a preset sliding window, and before acquiring the first key frame image, the method further includes:

[0031] Determining whether the number of key frame images in the preset sliding window meets a preset semantic map initialization condition;

[0032] If the conditions are met, a second key frame image is obtained from the preset sliding window, and a road surface key point in the second key frame image is determined, and a road surface three-dimensional point corresponding to the road surface key point in the second key frame image is determined based on high-precision positioning data corresponding to the second key frame image;

[0033] Determining the 3D road surface element points corresponding to the second key frame image using a preset oblique projection algorithm based on the 3D road surface points corresponding to the road surface key points in the second key frame image and the road surface semantic segmentation result corresponding to the second key frame image;

[0034] The semantic map is initialized according to the three-dimensional points of the road surface elements corresponding to the second key frame image to obtain an initialized semantic map.

[0035] In a second aspect, an embodiment of the present application further provides a semantic map construction device, wherein the device includes:

[0036] A first acquisition unit is configured to acquire a first key frame image, and a road surface semantic segmentation result and corresponding high-precision positioning data corresponding to the first key frame image;

[0037] a first determining unit, configured to determine a road surface key point in the first key frame image, and determine a road surface three-dimensional point corresponding to the road surface key point in the first key frame image based on high-precision positioning data corresponding to the first key frame image;

[0038] a second determining unit, configured to determine, based on a road semantic segmentation result corresponding to the first key frame image and the road 3D points, a road element 3D point corresponding to the first key frame image using a preset oblique projection algorithm;

[0039] The construction unit is used to construct a semantic map according to the three-dimensional points of the road surface elements to obtain a constructed semantic map.

[0040] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0041] processor; and

[0042] A memory arranged to store computer executable instructions, which when executed cause the processor to perform any of the methods described above.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes any of the aforementioned methods.

[0044] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: The semantic map construction method of the embodiments of the present application first obtains a first keyframe image, and the semantic segmentation results and corresponding high-precision positioning data corresponding to the first keyframe image; then determines the road key points in the first keyframe image, and determines the road surface three-dimensional points corresponding to the road key points in the first keyframe image based on the high-precision positioning data corresponding to the first keyframe image; then, based on the road semantic segmentation results and the three-dimensional road points corresponding to the first keyframe image, a preset oblique projection algorithm is used to determine the three-dimensional points of the road surface elements corresponding to the first keyframe image; finally, a semantic map is constructed based on the three-dimensional road surface element points to obtain a constructed semantic map. The semantic map construction method of the embodiments of the present application utilizes the road semantic segmentation results and high-precision positioning data to construct a semantic map based on road surface elements, is not affected by any degree of vehicle body undulation, and does not require pre-calibration of camera-to-ground extrinsic parameters. It is more applicable to different scenarios, has high mapping accuracy, consumes little computing power, and can achieve real-time mapping, providing lane-level positioning basis for navigation and positioning in autonomous driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 A flowchart of a method for constructing a semantic map according to an embodiment of the present application is shown;

[0047] Figure 2 This is a schematic diagram of the structure of a semantic map construction device in an embodiment of the present application;

[0048] Figure 3 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0051] The present application embodiment provides a method for constructing a semantic map, such as Figure 1As shown, a flowchart of a semantic map construction method in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S140:

[0052] Step S110: Acquire a first key frame image, and a road surface semantic segmentation result and corresponding high-precision positioning data corresponding to the first key frame image.

[0053] When constructing a semantic map, the embodiment of the present application needs to first obtain the first keyframe image. A "keyframe" can be understood as a relatively representative frame among all image frames, characterized by rich feature points, clear images, and uniform feature point distribution. To reduce storage space and computational complexity and improve system operation speed, only keyframes can be operated during the mapping process. Therefore, when a new image frame is acquired, the embodiment of the present application can first determine whether the image frame can be used as a keyframe for subsequent processing.

[0054] In addition, it is necessary to further obtain the road semantic segmentation results and corresponding high-precision positioning data corresponding to the first key frame image. The road semantic segmentation results are mainly the results obtained by segmenting the drivable area in the image and road elements such as lane lines, arrows, stop lines, and sidewalks. They can be obtained based on the existing semantic segmentation model based on deep learning training and are not specifically limited here.

[0055] High-precision positioning data can be obtained by relying on the high-precision combined navigation and positioning equipment on the vehicle. Since the data output frequency of the combined navigation and positioning equipment is generally higher than the output frequency of the image data, the high-precision positioning data can be time-synchronized with the image data. For example, the high-precision positioning data can be interpolated according to the synchronization time difference, thereby improving the mapping accuracy.

[0056] Step S120 , determining road surface key points in the first key frame image, and determining road surface three-dimensional points corresponding to the road surface key points in the first key frame image based on high-precision positioning data corresponding to the first key frame image.

[0057] After obtaining the first key frame image, it is also necessary to determine the road surface key points in the first key frame image. The key points here can be regarded as the two-dimensional pixel points corresponding to the road surface extracted from the first key frame image. Therefore, it is necessary to further combine the high-precision positioning data corresponding to the first key frame image to convert the road surface key points extracted from the first key frame image into three-dimensional road surface points in the world coordinate system, so as to construct a semantic map based on the three-dimensional road surface points.

[0058] Step S130 : determining the road surface element 3D points corresponding to the first key frame image using a preset oblique projection algorithm according to the road surface semantic segmentation result corresponding to the first key frame image and the road surface 3D points.

[0059] Based on the above steps, all the corresponding three-dimensional road points in the first key frame image can be determined, and these three-dimensional road points are obtained based on all the feature points on the road surface. In the subsequent actual positioning process, it is often necessary to provide road element information in the semantic map that is more obvious and easier to accurately match successfully, such as lane lines, arrows, stop lines, sidewalks, etc. Therefore, the embodiment of the present application needs to be based on the road semantic segmentation results and road three-dimensional points corresponding to the first key frame image, and use a preset oblique projection algorithm to further project the pixel points of the road elements segmented from the first key frame image onto the ground to obtain the three-dimensional road element points, that is, to determine the three-dimensional position information of each road element pixel point in the world coordinate system.

[0060] Step S140 : constructing a semantic map based on the three-dimensional points of the road surface elements to obtain a constructed semantic map.

[0061] After obtaining the three-dimensional points of the road surface elements corresponding to the first key frame image, map stitching can be performed based on the specific positions of the three-dimensional points of the road surface elements to obtain a constructed semantic map for subsequent positioning of the autonomous driving vehicle.

[0062] The semantic map construction method of the embodiment of the present application utilizes the road semantic segmentation results and high-precision positioning data to realize the construction of a semantic map based on road elements. It is not affected by any degree of vehicle body ups and downs, and does not require advance calibration of the external parameters from the camera to the ground. It is more applicable to different scenarios, has high mapping accuracy, consumes little computing power, and can achieve real-time mapping, providing a lane-level positioning basis for navigation and positioning in autonomous driving scenarios.

[0063] In some embodiments of the present application, determining the road surface key points in the first key frame image includes: performing histogram equalization processing on the first key frame image to obtain a processed first key frame image; using a preset tracking algorithm to determine the key points in the processed first key frame image; using a preset filtering strategy to filter the key points in the processed first key frame image to obtain filtered key points in the first key frame image as the road surface key points in the first key frame image.

[0064] When extracting road surface key points in the first key frame image, the embodiment of the present application can first perform histogram equalization on the first key frame image to reduce the impact of illumination changes, and then use a preset tracking algorithm such as an optical flow tracking algorithm to determine the key points in the processed first key frame image. The key points here include the key points of any target that may be of interest in the image, which may include both road surface key points and non-road surface key points. The semantic map ultimately to be constructed by the embodiment of the present application only needs to rely on road surface key points. Therefore, a certain filtering strategy can be used here to filter the extracted key points. For example, non-road surface key points can be filtered out, and key points with large tracking errors can also be filtered out, thereby ensuring that the extracted key points have high robustness and tracking accuracy.

[0065] In some embodiments of the present application, the use of a preset filtering strategy to filter the key points in the processed first key frame image to obtain the road surface key points in the filtered first key frame image includes: using an inverse optical flow tracking algorithm and a basic matrix algorithm to filter the key points in the processed first key frame image; and / or, generating a mask based on the area of interest in the processed first key frame image and the drivable area of the road surface in the processed first key frame image, and using the mask to filter the key points in the processed first key frame image.

[0066] On the one hand, when filtering the key points in the first key frame image, the embodiment of the present application can use the inverse optical flow tracking algorithm and the basic matrix algorithm to determine the matching error of the key points, and filter out the key points with large matching errors or tracking errors, thereby improving the tracking accuracy of the key points.

[0067] On the other hand, the embodiment of the present application can determine the drivable road area and the region of interest (ROI) in the first key frame image based on the road semantic segmentation results, and then generate a mask based on the segmented images of the region of interest and the drivable road area, and use the mask to filter out key points tracked outside the ground, thereby improving the robustness of the key points.

[0068] In addition, in order to ensure the subsequent mapping effect, the number of road surface key points in the first key frame image needs to meet the requirements of a preset number threshold, and the number of road surface key points remaining in the first key frame image after the above-mentioned filtering process may not meet the requirements of the preset number threshold. Therefore, the embodiment of the present application can extract the remaining road surface key points from the current frame monocular image based on the key points tracked by the optical flow tracking algorithm and the mask, that is, the road surface key points are extracted in the image area outside the road surface key points tracked by the optical flow and the mask, so as to ensure that the number of road surface key points in the first key frame image is not less than the preset number threshold.

[0069] In some embodiments of the present application, the first key frame image is located in a preset sliding window. Before obtaining the first key frame image, the method further includes: determining whether the number of frames of the key frame image in the preset sliding window meets the preset semantic map initialization condition; if so, obtaining the second key frame image from the preset sliding window, and determining the road surface key points in the second key frame image, and determining the road surface three-dimensional points corresponding to the road surface key points in the second key frame image based on the high-precision positioning data corresponding to the second key frame image; determining the road surface element three-dimensional points corresponding to the second key frame image based on the road surface three-dimensional points corresponding to the road surface key points in the second key frame image and the road surface semantic segmentation result corresponding to the second key frame image using a preset oblique projection algorithm; initializing the semantic map based on the road surface element three-dimensional points corresponding to the second key frame image to obtain the initialized semantic map.

[0070] The key frame images of the embodiment of the present application can be maintained by a preset sliding window. Each time a new image is acquired, it can be first determined whether it can be used as a key frame image. For example, it can be determined based on the size of the disparity between the image frames. For the first frame image, it can be directly used as a key frame image, and for the subsequently acquired images, it can be determined based on the size of the disparity between the image frames. For example, if the disparity between the new image and the previous key frame image is greater than the preset disparity threshold, it means that there is sufficient relative motion between the two frames of image, and then the new image can be stored as a key frame image in the preset sliding window, and at the same time, the key point extraction and semantic segmentation operations in the aforementioned embodiment are performed on the key frame image. The size N of the preset sliding window can be flexibly set according to actual needs and is not specifically limited here.

[0071] During the entire semantic mapping process, the semantic map can be initialized based on the preset sliding window. This process first determines whether the number of keyframe images within the preset sliding window has reached the preset sliding window size N. If so, initialization conditions are met, and a second keyframe image can be obtained from the preset sliding window. Since no 3D road point information is available during the initialization phase, the first several frames from the preset sliding window can be used as the second keyframe image. For example, keyframe images from frames 1 to N / 2 can be used for initialization to ensure mapping effectiveness during the initialization phase.

[0072] When initializing multiple second key frame images based on a preset sliding window, all road surface key points in each second key frame image can be first subjected to multi-frame triangulation processing to obtain the road surface three-dimensional points corresponding to each second key frame image, and the size of the reprojection error corresponding to the road surface three-dimensional point is determined. If the preset error threshold is not met, the road surface three-dimensional point is set as an invalid point.

[0073] A plane fitting algorithm is then used to perform plane fitting on each 3D road surface point corresponding to each second keyframe image, obtaining the corresponding plane fitting equation for each second keyframe image. Contour points are then extracted from the road surface element segmentation image corresponding to each second keyframe image, obtaining road surface element contour points. Each road surface element contour point is then transformed to the camera normalization plane and unit vectorized to obtain the corresponding unit vector. The equation of a spatial line in the camera coordinate system is then determined based on the corresponding unit vector and the camera optical center. The intersection of the spatial line equation and the plane fitting equation for each road surface element contour point is the 3D coordinate of the road surface element contour point on the ground, i.e., the 3D road surface element point. Finally, all obtained 3D road surface element points are added to the semantic map, completing the semantic map initialization process.

[0074] After the initialization process is completed, the subsequent mapping process can be carried out. That is, each time a new image frame is obtained, it can be first determined whether it can be used as a key frame image. If it can be used as a key frame image, it will be directly stored in the preset sliding window, and the earliest key frame image in the preset sliding window will be removed. Since the initialization stage only performs multi-frame triangulation and other processing on a portion of the key frame images in the preset sliding window, such as the key frame images of frames 1 to N / 2, after the newly obtained key frame image is stored in the preset sliding window, it means that the key frame images of frames N / 2 to N have not yet been processed. At this time, based on the results of the initialization processing, multi-frame triangulation and other processing can be performed on the key frame image of frame N / 2. Similarly, each subsequent key frame image is obtained, the key frame image of frame N / 2 in the preset sliding window is always processed, thereby achieving real-time mapping.

[0075] Therefore, what we want to make clear here is that the "first key frame image" defined in the embodiment of the present application is for the subsequent real-time mapping process, which can be understood as the earliest key frame image that has not been processed in the preset sliding window, and the "second key frame image" is for the initialization process, which can be understood as the first multiple key frame images in the preset sliding window used for initialization processing.

[0076] In some embodiments of the present application, determining the road surface three-dimensional points corresponding to the road surface key points in the first key frame image based on the high-precision positioning data corresponding to the first key frame image includes: determining the camera pose corresponding to the first key frame image based on the high-precision positioning data and the external parameters from the camera to the vehicle body; and determining the road surface three-dimensional points corresponding to the road surface key points in the first key frame image using a preset triangulation algorithm based on the camera pose corresponding to the first key frame image.

[0077] Since the high-precision positioning information corresponds to the vehicle's pose, the camera pose corresponding to the first keyframe image can be calculated based on the high-precision positioning data corresponding to the first keyframe image and the pre-calibrated camera-to-vehicle extrinsic parameter transformation relationship. Based on the camera pose corresponding to the first keyframe image, the multi-frame triangulation algorithm can be used to determine the 3D road surface points corresponding to the road surface key points in the first keyframe image. The multi-frame triangulation algorithm is the process of solving the position coordinates of 3D points in the world coordinate system when the 2D observation coordinates of multiple frames, the camera intrinsic parameter matrix, and the camera pose of each frame are known. The multi-frame algorithm here means n ≥ 2. If a road surface key point is observed more than twice in a row, then triangulation based on the road surface key points observed in these multiple frames will produce a road surface point with smaller errors and more reliable accuracy.

[0078] In addition, in the real-time mapping process, the embodiment of the present application can also determine the size of the reprojection error corresponding to the road surface three-dimensional point based on the triangulation result. If the reprojection error exceeds the preset error threshold, it means that the position information of the road surface three-dimensional point is not reliable enough. In order to ensure the mapping accuracy, the road surface three-dimensional point can be marked as an invalid point for subsequent deletion.

[0079] In some embodiments of the present application, determining the three-dimensional points of the road surface elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road surface semantic segmentation result and the road surface three-dimensional points corresponding to the first key frame image includes: performing plane fitting on the road surface three-dimensional points corresponding to the first key frame image to obtain the plane fitting result corresponding to the first key frame image; determining the three-dimensional points of the road surface elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road surface semantic segmentation result and the plane fitting result corresponding to the first key frame image.

[0080] In the embodiment of the present application, when determining the three-dimensional points of the road surface elements corresponding to the first key frame image, a preset plane fitting algorithm, such as the RANSAC (Random Sample Consensus) algorithm, may be used to perform plane fitting on all three-dimensional points of the road surface corresponding to the first key frame image, thereby obtaining a plane fitting equation, which may be expressed as follows, for example:

[0081] Ax+By+Cz+D=0;

[0082] Among them, (x, y, z) is the three-dimensional position coordinate of the three-dimensional point on the road surface, and A, B, C, and D are the coefficients of the plane fitting equation.

[0083] The above process is equivalent to determining a spatial plane through multiple three-dimensional road surface points. Then, combined with the semantic segmentation results such as the road surface elements corresponding to the first key frame image, the three-dimensional points of the road surface elements corresponding to the first key frame image can be determined using the preset oblique projection algorithm.

[0084] In some embodiments of the present application, the plane fitting result includes a plane fitting equation, the road surface semantic segmentation result includes a road surface element segmentation image, and the determining of the road surface element three-dimensional points corresponding to the first key frame image using a preset oblique projection algorithm based on the road surface semantic segmentation result corresponding to the first key frame image and the plane fitting result includes: performing contour point extraction on the road surface element segmentation image to obtain road surface element contour points; converting the road surface element contour points to a camera normalized plane and performing unit vectorization processing to obtain unit vectors corresponding to the road surface element contour points; determining the spatial straight line equation of the unit vector corresponding to the road surface element contour point in the camera coordinate system based on the unit vector corresponding to the road surface element contour point and the camera optical center; and determining the road surface element three-dimensional points corresponding to the first key frame image based on the intersection of the spatial straight line equation of the unit vector corresponding to the road surface element contour point in the camera coordinate system and the plane fitting equation.

[0085] In the embodiments of the present application, when using a preset oblique projection algorithm to determine the three-dimensional points of road surface elements corresponding to the first keyframe image, contour points can be first extracted from the binary image of the road surface elements obtained through semantic segmentation, thereby obtaining contour points of all road surface elements, such as lane lines, arrows, and stop lines. These road surface element contour points are located in a pixel coordinate system. Therefore, these road surface element contour points can be first converted to the camera normalized plane using camera intrinsic parameters. The conversion to the camera normalized plane involves projecting the coordinates P(X, Y, Z) of the road surface element contour points in the camera coordinate system onto the normalized plane Z = 1, thereby obtaining the normalized coordinates P'(X / Z, Y / Z, 1) of the road surface element contour points. To facilitate subsequent processing, the normalized coordinates of each road surface element contour point can be further unit-vectorized, that is, converted into a vector with a defined direction and a modulus of 1.

[0086] Since the camera optical center is the origin of the camera coordinate system (0,0,0), the spatial straight line equation in the camera coordinate system can be calculated based on the unit vector corresponding to each road surface element contour point and the camera optical center. By solving the intersection of the spatial straight line equation corresponding to each road surface element contour point and the plane fitting equation obtained by the above fitting, the three-dimensional coordinates of each road surface element contour point on the ground can be obtained.

[0087] In some embodiments of the present application, the first key frame image is located in a preset sliding window. After constructing a semantic map based on the three-dimensional points of the road surface elements and obtaining the constructed semantic map, the method further includes: determining whether the three-dimensional points of the road surface elements corresponding to each key frame image in the preset sliding window meet a preset deletion condition; if so, deleting the three-dimensional points of the road surface elements from the preset sliding window.

[0088] After completing the above processing of each key frame image, the 3D road surface points corresponding to all the current key frame images in the preset sliding window can be deleted to save the occupied memory while ensuring the subsequent mapping accuracy.

[0089] The deletion strategy here may include, for example, whether the landmark 3D point is an invalid point. If it is an invalid point, it can be deleted directly. Or it may include whether the landmark 3D point has no effect on the processing of the first key frame image and subsequent image frames, that is, whether the landmark 3D point has no longer been observed in the first key frame image and subsequent image frames. If so, it can be deleted directly.

[0090] Based on the above embodiments, the semantic mapping process of the present application is mainly divided into an initialization process and a real-time mapping process. The main difference between the two is that different key frame images are selected. The similarities between the two are the determination of the key points of the key frame images and the triangulation processing, plane fitting, and preset oblique projection processes.

[0091] In summary, the semantic map construction method of this application has achieved at least the following technical effects:

[0092] 1) It is not affected by any degree of vehicle body ups and downs and is more robust than the mapping solution based on the IPM algorithm in uneven road scenes;

[0093] 2) No need to calibrate the external parameters from the camera to the ground in advance, making it more applicable to different scenarios;

[0094] 3) High mapping accuracy, low computing power consumption, and real-time mapping.

[0095] The embodiment of the present application also provides a semantic map construction device 200, such as Figure 2As shown, a schematic diagram of the structure of a semantic map construction device in an embodiment of the present application is provided. The device 200 includes: a first acquisition unit 210, a first determination unit 220, a second determination unit 230 and a construction unit 240, wherein:

[0096] A first acquisition unit 210 is configured to acquire a first key frame image, and a road semantic segmentation result and corresponding high-precision positioning data corresponding to the first key frame image;

[0097] A first determining unit 220 is configured to determine a road surface key point in the first key frame image, and determine a road surface three-dimensional point corresponding to the road surface key point in the first key frame image based on high-precision positioning data corresponding to the first key frame image;

[0098] A second determining unit 230 is configured to determine the three-dimensional points of road surface elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road surface semantic segmentation result corresponding to the first key frame image and the three-dimensional points of the road surface;

[0099] The construction unit 240 is configured to construct a semantic map according to the three-dimensional points of the road surface elements to obtain a constructed semantic map.

[0100] In some embodiments of the present application, the first determination unit 220 is specifically used to: perform histogram equalization processing on the first key frame image to obtain a processed first key frame image; use a preset tracking algorithm to determine the key points in the processed first key frame image; use a preset filtering strategy to filter the key points in the processed first key frame image to obtain filtered key points in the first key frame image as road surface key points in the first key frame image.

[0101] In some embodiments of the present application, the first determination unit 220 is specifically used to: filter the key points in the processed first key frame image using the inverse optical flow tracking algorithm and the basic matrix algorithm; and / or generate a mask based on the area of interest in the processed first key frame image and the drivable area of the road in the processed first key frame image, and use the mask to filter the key points in the processed first key frame image.

[0102] In some embodiments of the present application, the first determination unit 220 is specifically used to: determine the camera pose corresponding to the first key frame image based on the high-precision positioning data and the external parameters from the camera to the vehicle body; based on the camera pose corresponding to the first key frame image, determine the three-dimensional road surface points corresponding to the road surface key points in the first key frame image using a preset triangulation algorithm.

[0103] In some embodiments of the present application, the second determination unit 230 is specifically used to: perform plane fitting on the three-dimensional points of the road surface corresponding to the first key frame image to obtain the plane fitting result corresponding to the first key frame image; and determine the three-dimensional points of the road surface elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road surface semantic segmentation result and the plane fitting result corresponding to the first key frame image.

[0104] In some embodiments of the present application, the plane fitting result includes a plane fitting equation, the road surface semantic segmentation result includes a road surface element segmentation image, and the second determination unit 230 is specifically used to: extract contour points from the road surface element segmentation image to obtain road surface element contour points; convert the road surface element contour points to a camera normalized plane and perform unit vectorization processing to obtain unit vectors corresponding to the road surface element contour points; determine the spatial straight line equation of the unit vector corresponding to the road surface element contour point in the camera coordinate system based on the unit vector corresponding to the road surface element contour point and the camera optical center; determine the road surface element three-dimensional point corresponding to the first key frame image based on the intersection of the spatial straight line equation of the unit vector corresponding to the road surface element contour point in the camera coordinate system and the plane fitting equation.

[0105] In some embodiments of the present application, the first key frame image is located in a preset sliding window, and the device further includes: a third determination unit, used to determine whether the three-dimensional points of the road surface elements corresponding to each key frame image in the preset sliding window meet a preset deletion condition; and a deletion unit, used to delete the three-dimensional points of the road surface elements from the preset sliding window if the condition is met.

[0106] In some embodiments of the present application, the first key frame image is located in a preset sliding window, and the device further includes: a fourth determination unit, used to determine whether the number of frames of the key frame image in the preset sliding window meets the preset semantic map initialization condition; a second acquisition unit, used to obtain the second key frame image from the preset sliding window if it meets the condition, and determine the road surface key points in the second key frame image, and determine the road surface three-dimensional points corresponding to the road surface key points in the second key frame image according to the high-precision positioning data corresponding to the second key frame image; a fifth determination unit, used to determine the road surface element three-dimensional points corresponding to the second key frame image according to the road surface three-dimensional points corresponding to the road surface key points in the second key frame image and the road surface semantic segmentation result corresponding to the second key frame image using a preset oblique projection algorithm; an initialization unit, used to initialize the semantic map according to the road surface element three-dimensional points corresponding to the second key frame image to obtain an initialized semantic map.

[0107] It can be understood that the above-mentioned semantic map construction device can implement each step of the semantic map construction method provided in the above-mentioned embodiment. The relevant explanations about the semantic map construction method are applicable to the semantic map construction device and will not be repeated here.

[0108] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0109] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0110] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0111] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a semantic map construction device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0112] Acquire a first key frame image, and a road semantic segmentation result and corresponding high-precision positioning data corresponding to the first key frame image;

[0113] Determining a road surface key point in the first key frame image, and determining a road surface three-dimensional point corresponding to the road surface key point in the first key frame image based on high-precision positioning data corresponding to the first key frame image;

[0114] Determining the 3D points of road elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road semantic segmentation result corresponding to the first key frame image and the 3D points of the road surface;

[0115] A semantic map is constructed according to the three-dimensional points of the road surface elements to obtain a constructed semantic map.

[0116] The above application Figure 1 The methods performed by the semantic map construction device disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0117] The electronic device may also perform Figure 1 The method executed by the semantic map construction device in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0118] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1The method performed by the semantic map construction device in the embodiment shown is specifically used to perform:

[0119] Acquire a first key frame image, and a road semantic segmentation result and corresponding high-precision positioning data corresponding to the first key frame image;

[0120] Determining a road surface key point in the first key frame image, and determining a road surface three-dimensional point corresponding to the road surface key point in the first key frame image based on high-precision positioning data corresponding to the first key frame image;

[0121] Determining the 3D points of road elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road semantic segmentation result corresponding to the first key frame image and the 3D points of the road surface;

[0122] A semantic map is constructed according to the three-dimensional points of the road surface elements to obtain a constructed semantic map.

[0123] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0127] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0128] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0129] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0130] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0131] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A semantic map construction method, wherein: The method comprises: Acquire a first key frame image, and a road semantic segmentation result and corresponding high-precision positioning data corresponding to the first key frame image; Determining a road surface key point in the first key frame image, and determining a road surface three-dimensional point corresponding to the road surface key point in the first key frame image based on high-precision positioning data corresponding to the first key frame image; Determining the 3D points of road elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road semantic segmentation result corresponding to the first key frame image and the 3D points of the road surface; Constructing a semantic map based on the three-dimensional points of the road surface elements to obtain a constructed semantic map; The first key frame image is located in a preset sliding window. Before acquiring the first key frame image, the method further includes: Determining whether the number of key frame images in the preset sliding window meets a preset semantic map initialization condition; If the conditions are met, a second key frame image is obtained from the preset sliding window, and a road surface key point in the second key frame image is determined, and a road surface three-dimensional point corresponding to the road surface key point in the second key frame image is determined based on high-precision positioning data corresponding to the second key frame image; Determining the 3D road surface element points corresponding to the second key frame image using a preset oblique projection algorithm based on the 3D road surface points corresponding to the road surface key points in the second key frame image and the road surface semantic segmentation result corresponding to the second key frame image; The semantic map is initialized according to the three-dimensional points of the road surface elements corresponding to the second key frame image to obtain an initialized semantic map.

2. The method according to claim 1, wherein: Determining the road surface key points in the first key frame image includes: performing histogram equalization processing on the first key frame image to obtain a processed first key frame image; Determining key points in the processed first key frame image using a preset tracking algorithm; The key points in the processed first key frame image are filtered using a preset filtering strategy to obtain filtered key points in the first key frame image as road surface key points in the first key frame image.

3. The method according to claim 1, wherein: Determining the road surface three-dimensional point corresponding to the road surface key point in the first key frame image according to the high-precision positioning data corresponding to the first key frame image includes: Determining a camera pose corresponding to the first key frame image based on the high-precision positioning data and external parameters from the camera to the vehicle body; Based on the camera pose corresponding to the first key frame image, a preset triangulation algorithm is used to determine a road surface three-dimensional point corresponding to a road surface key point in the first key frame image.

4. The method according to claim 1, wherein: The determining, based on the road surface semantic segmentation result corresponding to the first key frame image and the road surface three-dimensional points, using a preset oblique projection algorithm to determine the road surface element three-dimensional points corresponding to the first key frame image comprises: Performing plane fitting on the three-dimensional points of the road surface corresponding to the first key frame image to obtain a plane fitting result corresponding to the first key frame image; According to the road surface semantic segmentation result corresponding to the first key frame image and the plane fitting result, a preset oblique projection algorithm is used to determine the three-dimensional points of the road surface elements corresponding to the first key frame image.

5. The method according to claim 4, wherein: The plane fitting result includes a plane fitting equation, the road surface semantic segmentation result includes a road surface element segmentation image, and determining the three-dimensional points of the road surface elements corresponding to the first key frame image using a preset oblique projection algorithm based on the road surface semantic segmentation result corresponding to the first key frame image and the plane fitting result includes: Extracting contour points from the road surface element segmentation image to obtain road surface element contour points; Converting the pavement element contour points to a camera normalized plane and performing unit vectorization processing to obtain unit vectors corresponding to the pavement element contour points; Determining a spatial straight line equation of the unit vector corresponding to the pavement element contour point in a camera coordinate system based on the unit vector corresponding to the pavement element contour point and the camera optical center; The three-dimensional point of the road surface element corresponding to the first key frame image is determined according to the intersection of the spatial straight line equation of the unit vector corresponding to the road surface element contour point in the camera coordinate system and the plane fitting equation.

6. The method of claim 1, wherein: The first key frame image is located in a preset sliding window. After constructing a semantic map based on the three-dimensional points of the road surface elements to obtain the constructed semantic map, the method further includes: Determining whether the three-dimensional points of road surface elements corresponding to each key frame image in the preset sliding window meet a preset deletion condition; If satisfied, the road surface element three-dimensional point is deleted from the preset sliding window.

7. A semantic map construction device, wherein: The device comprises: A first acquisition unit is configured to acquire a first key frame image, and a road surface semantic segmentation result and corresponding high-precision positioning data corresponding to the first key frame image; a first determining unit, configured to determine a road surface key point in the first key frame image, and determine a road surface three-dimensional point corresponding to the road surface key point in the first key frame image based on high-precision positioning data corresponding to the first key frame image; a second determining unit, configured to determine, based on a road semantic segmentation result corresponding to the first key frame image and the road 3D points, a road element 3D point corresponding to the first key frame image using a preset oblique projection algorithm; A construction unit, configured to construct a semantic map based on the three-dimensional points of the road surface elements to obtain a constructed semantic map; The first key frame image is located in a preset sliding window, and the device further includes: A fourth determining unit, configured to determine whether the number of key frame images in the preset sliding window satisfies a preset semantic map initialization condition; a second acquisition unit, configured to acquire a second key frame image from the preset sliding window if the condition is satisfied, determine a road surface key point in the second key frame image, and determine a road surface three-dimensional point corresponding to the road surface key point in the second key frame image based on high-precision positioning data corresponding to the second key frame image; a fifth determining unit, configured to determine, based on the three-dimensional road surface points corresponding to the road surface key points in the second key frame image and the road surface semantic segmentation result corresponding to the second key frame image, the three-dimensional road surface element points corresponding to the second key frame image using a preset oblique projection algorithm; An initialization unit is used to initialize a semantic map according to the three-dimensional points of road surface elements corresponding to the second key frame image to obtain an initialized semantic map.

8. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute the method according to any one of claims 1 to 6.

Images