Method and apparatus for constructing three-dimensional map, electronic device, and storage medium

Abstract

The application relates to the technical field of maps, and discloses a method and device for constructing a three-dimensional map, electronic equipment and a storage medium, the method comprising the following steps: calculating a transfer matrix between road elements matched with feature points in a first road image and a second road image; performing semantic association on road elements in the first road image and road elements in the second road image according to the transfer matrix corresponding to the road elements matched with the feature points; determining three-dimensional space information corresponding to the road elements associated semantically according to position information of a plurality of feature points corresponding to the road elements associated semantically in the first road image and position information of the plurality of feature points in the second road image; and constructing a three-dimensional map according to the three-dimensional space information of the road elements associated semantically and an element type to which the road elements associated semantically belong. The scheme can reduce the cost of constructing a three-dimensional map and reduce the requirement for data collection.

Description

Technical Field

[0001] This application relates to the field of map technology, and more specifically, to a method, apparatus, electronic device, and storage medium for constructing three-dimensional maps. Background Technology

[0002] Building 3D maps using computer vision technology is a crucial task in map-related operations, with wide applications in road production automation, high-precision map construction, and autonomous driving. Common technologies for this include extensive road image matching and point cloud data collected by LiDAR (Light Detection and Ranging) to construct 3D maps. This method, relying on LiDAR point cloud data, has high data acquisition requirements and necessitates processing of both road images and point cloud data, resulting in a large data processing volume. The higher the map update frequency, the greater the data processing workload for 3D map construction. In summary, building 3D maps using a combination of image matching and LiDAR is costly. Summary of the Invention

[0003] In view of the above problems, this application proposes a method, apparatus, electronic device and storage medium for constructing three-dimensional maps to solve the problem of high cost of three-dimensional map construction in related technologies.

[0004] According to one aspect of the embodiments of this application, a method for constructing a three-dimensional map is provided. The method includes: calculating a transition matrix between road elements in a first road image and feature points in a second road image; the first road image and the second road image are different road images acquired from two acquisition points with an interval distance not exceeding a distance threshold; semantically associating road elements in the first road image and road elements in the second road image based on the transition matrix corresponding to the road elements with matching feature points, wherein the two semantically associating road elements represent the same road element in space; determining three-dimensional spatial information corresponding to the semantically associating road elements based on the position information of multiple feature points corresponding to the semantically associating road elements in the first road image and the position information of multiple feature points in the second road image; and constructing a three-dimensional map based on the three-dimensional spatial information of the semantically associating road elements and the element type to which the semantically associating road elements belong.

[0005] According to one aspect of the embodiments of this application, an apparatus for constructing a three-dimensional map is provided. The apparatus includes: a transition matrix determination module, configured to calculate a transition matrix between road elements in a first road image and feature points in a second road image; the first road image and the second road image are different road images acquired from two acquisition points with an interval not exceeding a distance threshold; a semantic association module, configured to semantically associate road elements in the first road image and road elements in the second road image according to the transition matrix corresponding to the road elements whose feature points are matched, wherein the two semantically associated road elements represent the same road element in space; a three-dimensional spatial information determination module, configured to determine the three-dimensional spatial information corresponding to the semantically associated road element based on the position information of multiple feature points corresponding to the semantically associated road element in the first road image and the position information of the feature points in the second road image; and a map construction module, configured to construct a three-dimensional map based on the three-dimensional spatial information of the semantically associated road element and the element type to which the semantically associated road element belongs.

[0006] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method for constructing a three-dimensional map as described above.

[0007] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, implement the method for constructing a three-dimensional map as described above.

[0008] According to one aspect of the embodiments of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the method for constructing a three-dimensional map as described above.

[0009] In this application, after feature point matching of the first and second road images, a transition matrix is ​​calculated between road elements in the first and second road images whose feature points match. The road elements in the different road images are then semantically associated based on the transition matrix. Subsequently, the 3D spatial information of the road elements is determined based on their positional information in the two road images, and a 3D map is constructed based on this information. This process utilizes road images acquired from two different perspectives to construct a 3D map without relying on LiDAR for point cloud data acquisition. This eliminates the dependence on LiDAR equipment for 3D map construction and reduces the cost of 3D map construction. Furthermore, this solution allows for timely 3D map updates based on updated road images when roads are being updated.

[0010] Furthermore, in the scheme of this application, based on the feature point extraction of two road images, not only are feature points in different road images matched, but also the road elements in the two road images are further semantically associated based on the transition matrix. Compared with the method of associating road elements only based on feature point matching, this scheme references more information, thereby ensuring the accuracy of the semantic association of road elements, ensuring the accuracy of the three-dimensional spatial information of the determined road elements, and thus ensuring the accuracy of the generated three-dimensional map. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of this application according to an embodiment of this application.

[0013] Figure 2 This is a flowchart illustrating a method for constructing a three-dimensional map according to an embodiment of this application.

[0014] Figure 3A An example diagram illustrates the transformation of pixels in an image using a homography matrix.

[0015] Figure 3B A schematic diagram illustrating the mapping relationship between exemplary pixels and epipolar lines.

[0016] Figure 4 An exemplary diagram illustrates the clustering of multiple road elements in a road image.

[0017] Figure 5 This is a schematic diagram illustrating the expansion of an aggregate group according to an embodiment of this application.

[0018] Figure 6 yes Figure 2 Step 220 in one embodiment is shown in a flowchart.

[0019] Figure 7 yes Figure 2 The flowchart of step 220 in the corresponding embodiment is shown in another embodiment.

[0020] Figure 8 A schematic diagram of the pinhole imaging principle is shown.

[0021] Figure 9An exemplary diagram illustrates the projection of a road trajectory from a first acquisition point onto the road image corresponding to the first road image.

[0022] Figure 10 This is a schematic diagram illustrating the division of a crossroads according to an embodiment of this application.

[0023] Figure 11 This is a flowchart illustrating a method for constructing a three-dimensional map according to another embodiment of this application.

[0024] Figure 12 This is a schematic diagram of road elements in a road image according to an embodiment of this application.

[0025] Figure 13 This is a schematic diagram illustrating the filtering of feature points in a road image according to an embodiment of this application.

[0026] Figure 14 This is a flowchart illustrating a method for constructing a three-dimensional map according to a specific embodiment of this application.

[0027] Figure 15 This is a schematic diagram of the structure of a multi-task parsing network according to an embodiment of this application.

[0028] Figure 16 This is a block diagram of an apparatus for constructing a three-dimensional map according to an embodiment of this application.

[0029] Figure 17 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0030] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0031] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0034] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using image acquisition devices and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0035] Constructing 3D maps using computer vision technology is a crucial task in map-related operations, with wide applications in road production automation, high-precision map building, and autonomous driving. Common technologies involve constructing 3D maps through extensive road image matching and point cloud data acquired by LiDAR. This method, relying on LiDAR point cloud data, has high data acquisition requirements and requires processing of both road images and point cloud data, resulting in a large data processing volume. Increased map update frequency leads to a corresponding increase in the data processing volume for 3D map construction. In summary, constructing 3D maps using a combination of image matching and LiDAR is costly. Based on this, the solution proposed in this application is presented.

[0036] Figure 1 This is a schematic diagram illustrating an application scenario of this application according to an embodiment of this application, such as... Figure 1As shown, this application scenario includes multiple terminals 110 and a server 120 that communicates with each terminal 110 via wired or wireless network. The terminal 110 is an electronic device with image acquisition capabilities, which can be a vehicle equipped with a dashcam, camera, smartphone, tablet, smart voice interaction device, in-vehicle terminal, aircraft, etc.

[0037] The terminal 110 can capture images of the road environment while moving along the travel trajectory on the road, thereby obtaining road images on the travel trajectory; for example, during vehicle travel, it can capture images at intervals through its driving recorder or other vehicle-mounted terminals with image acquisition capabilities to obtain multiple road images on the corresponding travel trajectory.

[0038] During the road image acquisition process, the terminal 110 can acquire the location information (such as latitude and longitude coordinates, such as GPS (Global Positioning System) information) of the terminal 110 when the road image is captured, and associate and store the road image with the location information of the terminal 110 when the image is captured, so as to use it for subsequent 3D map construction.

[0039] Because vehicles cover a wide area, using image acquisition devices within the vehicle can ensure comprehensive road image coverage, thereby providing more complete road images for constructing 3D maps.

[0040] Terminal 110 can send multiple collected road images to server 120, whereby server 120 can construct a 3D map according to the method of this application. Specifically, server 120 can construct a 3D map according to the following process: determining the transition matrix between matching road elements in two road images; semantically associating road elements in two road images based on the transition matrix; determining the 3D spatial information of the road element based on the feature point pairs in the semantically associating road element; and constructing a 3D map based on the 3D spatial information of the road element.

[0041] The server 120 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0042] The solutions in the embodiments of this application can be applied to the technical fields of 3D maps, navigation, autonomous driving, intelligent vehicle control, vehicle networking, intelligent transportation, driver assistance, and cloud computing.

[0043] The implementation details of the technical solutions in the embodiments of this application are described in detail below.

[0044] Figure 2 This is a flowchart illustrating a method for constructing a 3D map according to an embodiment of this application. The method can be performed by… Figure 1 The server-side component 120 is executed. (Refer to...) Figure 2 As shown, the method includes at least steps 210 to 240, which are described in detail below:

[0045] Step 210: Calculate the transition matrix between road elements in the first road image that match feature points in the second road image; the first road image and the second road image are different road images acquired from two acquisition points with an interval distance not exceeding a distance threshold.

[0046] The first and second road images can be obtained by capturing images of the road environment using an image acquisition device during the journey. Specifically, the image acquisition device can be a dashcam in the vehicle, and the vehicle can capture images at intervals while driving on the road. The image acquisition device can also be a smartphone, tablet, camcorder, camera, etc., without specific limitations.

[0047] The first and second road images can be different road images collected at the same location point along the same driving trajectory, or at two location points with an interval of less than a distance threshold. Alternatively, they can be different road images collected at two location points with an interval of less than a distance threshold along different driving trajectories. In this way, the first and second road images contain a large number of identical road elements, which can improve the efficiency of 3D map construction.

[0048] Road elements refer to elements used to identify the road environment. Road elements can include traffic lights (such as traffic lights), speed limit signs, speed cameras, hazard signs, road name signs, no-entry signs, U-turn signs, etc.

[0049] Before step 210, target detection can be performed on the first road image and the second road image respectively, so as to determine the road elements included in the first road image and the position information of each road element in the first road image, and to determine the road elements included in the second road image and the position information of each road element in the second road image.

[0050] In some embodiments, object detection models can be used to detect objects in road images. These object detection models are constructed using one or more neural networks and can be used to identify road elements included in the road image, as well as the location information of these road elements within the image. In a specific embodiment, the object detection model can, based on the input road image, output the element type of each road element and the location information of the corresponding detection box. The detection box can be a bounding box (e.g., a rectangle) that encloses the road element, and the area enclosed by the detection box is considered the region where the road element is located.

[0051] As described above, road elements can be traffic lights, speed limit signs, electronic eyes, etc. Road elements can be pre-classified into various element types, such as element types for traffic lights, element types for speed limit signs, element types for e-cigarettes, element types for hazard signs, road types for road name signs, element types for no-entry signs, etc.

[0052] To ensure the accuracy of the location information of the detection boxes and the element types of road elements output by the object detection model, the model needs to be trained in advance using first training data. The first training data includes multiple first training images and corresponding first label information for each first training image. The first label information indicates the location information and element type of each road element included in the corresponding first training image. Then, the first training images are input into the object detection model, which performs road element detection and recognition, and inputs corresponding road element recognition information. This road element recognition information indicates the predicted location information and predicted element type of each road element in the predicted first training image. Subsequently, based on the road element recognition information and corresponding first label information of the first training images, the function value of the first loss function corresponding to the object detection model is calculated. The parameters of the object detection model are then adjusted in reverse based on the calculated function value until a first training termination condition is met. This first training termination condition can be the convergence of the first loss function or the number of iterations reaching a threshold, etc.

[0053] In a specific embodiment, feature points can be extracted from the first road image and the second road image. Then, feature points in the first road image and feature points in the second road image are associated. Subsequently, road elements in the first road image that match feature points in the second road image are determined based on the feature point association results. Specifically, if N feature points in road element A in the first road image are associated with feature points in road element B in the second road image, and if N is greater than a threshold, then road element A in the first road image and road element B in the second road image are determined as road elements with matched feature points.

[0054] Feature points can be points in an image where the grayscale value changes significantly or points on the image edge with large curvature (such as the intersection of two edges). During the feature point extraction process, a feature descriptor corresponding to the feature point is obtained. This feature descriptor can be considered as the identity authentication of the feature point. It is a vector with a fixed length, or in other words, a feature descriptor is a feature description vector used for the feature point.

[0055] Generally, feature points of the same spatial point under different viewpoints have highly similar feature descriptors. Therefore, feature points in the first road image and feature points in the second road image can be associated based on the feature descriptors corresponding to the feature points. Specifically, for feature point I in the first road image and feature point II in the second road image, if the similarity between the feature descriptor corresponding to feature point I and the feature descriptor corresponding to feature point II exceeds a similarity threshold, then feature point I in the first road image and feature point II in the second road image are determined to be associated feature points or matching feature points.

[0056] The transition matrix corresponding to the road element with matched feature points is used to describe the positional transformation relationship between multiple feature points in the road element in the first road image and multiple feature points in the corresponding road element in the second road image. That is, if road element A in the first road image and road element B in the second road image are road elements with matched feature points, then the corresponding transition matrix is ​​used to describe the positional transformation relationship between multiple feature points included in road element A in the first road image and multiple associated feature points included in road element B in the second road image.

[0057] The transition matrix can be either a homography matrix or a fundamental matrix. The homography matrix constrains the two-dimensional homogeneous coordinates of the same point in physical space across two pixel planes. In other words, the homography matrix describes the constraint relationship between pixels on two different pixel planes, and can be used to determine the position of the corresponding pixel in another image.

[0058] Figure 3AAn exemplary diagram illustrates the transformation of pixels in an image using a homography matrix. For example, pixel C1 in the first image can be transformed into pixel C2 in the second image using a homography matrix, pixel D1 in the first image can be transformed into pixel D2 in the second image using a homography matrix, and pixel E1 in the first image can be transformed into pixel E2 in the second image using a homography matrix.

[0059] Conversely, if the positional information of multiple sets of corresponding pixels in two images is known, the corresponding homography matrix can be determined in reverse. Typically, a 3x3 homography matrix can be determined using the positional information of four sets of associated feature point pairs. In other embodiments, the findHomography() function in the OpenCV package can also be used to output the corresponding homography matrix based on the positional information of multiple sets of associated feature point pairs.

[0060] The fundamental matrix describes the epipolar constraints of a 3D scene. It is independent of the structure of the 3D scene and depends only on the intrinsic and extrinsic parameters of the camera, requiring rotation and translation between the two cameras. It describes the constraints between points and lines; that is, the fundamental matrix represents the mapping relationship between an image point and the epipolar line in another image.

[0061] Figure 3B A schematic diagram illustrating the mapping relationship between pixels and epipolar lines is shown. In this diagram, plane P2 is the imaging plane corresponding to plane P1 when the image acquisition device is positioned O1, and plane P3 is the imaging plane corresponding to plane P1 when the image acquisition device is positioned O2. That is, O1 and O2 are the camera centers of the image acquisition device, respectively. The line connecting the camera centers O1 and O2 is called the baseline. e1 and e2 are the intersection points of the baseline with planes P2 and P3, respectively. e1 is the pixel point of O2 on plane P2 when the image acquisition device is positioned O1 and capturing plane P1, and e2 is the pixel point of O1 on plane P2 when the image acquisition device is positioned O2 and capturing plane P1.

[0062] Where x1 is the pixel point corresponding to point A1 in plane P1 on plane P2, and x2 is the pixel point corresponding to point A1 in plane P1 on plane P3. The epipolar line is the intersection of the epipolar plane and the imaging plane. The epipolar plane is a plane passing through the baseline, and e1, e2, O1, O2, x1, x2, and A1 are in the same plane.

[0063] From the above information, we know that for plane P1, which is obtained from different viewpoints as plane P2 and plane P3, any pixel in plane P2 has a corresponding epipolar line in plane P3. For example, pixel x2 in plane P2 corresponds to epipolar line l1 in plane P3, i.e., Fx2 = l1. Here, F is the fundamental matrix, representing the mapping relationship between pixel x2 and epipolar line l1 in plane P3.

[0064] For point A2 in three-dimensional space, it lies on the same straight line as O1 and A1. Therefore, when the image acquisition device captures plane P1 at position O1, the pixel points of A1 and A2 on plane P2 are the same, but the image depth is different. Based on Figure B in Figure 3, it can be seen that when the image acquisition device captures plane P1 at position O2, the pixel points x2 and x3 of A1 and A2 on plane P2 are still located on the same pair of epipolar lines.

[0065] Conversely, based on the images obtained from two perspectives in physical space, the corresponding fundamental matrix can be calculated according to the position information of the corresponding feature point pairs in the corresponding images.

[0066] The fundamental matrix F is also a 3×3 matrix, containing the spatial geometric relationship between the two images of the stereo pair during capture, as well as camera calibration parameters, including rotation, translation, principal point coordinates, and focal length. Typically, the fundamental matrix can be determined using the positional information of seven corresponding feature point pairs in the corresponding images. Alternatively, the fundamental matrix can be calculated directly using the `findFundamentalMat()` function in the OpenCV package. In specific embodiments, when the image acquisition devices at two different viewpoints only rotate without translating, the homography matrix can be used to describe the transformation relationship between the two images.

[0067] It is understood that in this application, the transition matrix determined based on the road elements can be considered a local transition matrix, because it does not correspond to the global transformation relationship from the first road image to the second road image, but rather reflects the transformation relationship between the two road elements associated with the feature points.

[0068] In some embodiments, step 210 includes: determining multiple feature point pairs corresponding to the road elements whose feature points match based on the road elements whose feature points match in the first road image and the second road image, wherein each feature point pair includes a first feature point in the road elements whose feature points match in the first road image and a second feature point in the road elements whose feature points match in the second road image; and determining a transition matrix corresponding to the road elements whose feature points match based on the position information of the first feature point and the position information of the second feature point in the multiple feature point pairs.

[0069] For example, if road element A in the first road image and road element B in the second road image are road elements whose feature points match, then the position information of K pairs of feature points located in road element A in the first road image and road element B in the second road image is used to determine the corresponding transition matrix; as described above, K is greater than or equal to 4.

[0070] It is understood that the above embodiments are applicable when there are at least 4 associated feature point pairs between road elements that match feature points; if there are less than 4 associated feature point pairs between road elements, the transition matrix can be determined according to the following embodiments.

[0071] In some other embodiments, step 210 includes: for a first target road element in the first road image to which the transition matrix is ​​to be calculated, determining the first target aggregation group to which the first target road element belongs; determining the transition matrix between the first target aggregation group and the second target aggregation group based on multiple feature point pairs that have a corresponding relationship in the first target aggregation group and the second target aggregation group, wherein the second target aggregation group refers to the aggregation group in the second road image that corresponds to the first target aggregation group; and using the transition matrix between the first target aggregation group and the second target aggregation group as the transition matrix between the first target road element in the first road image and the second target road element in the second road image; wherein the second target road element refers to the road element in the second road image whose feature point is located in the second target aggregation group.

[0072] In this embodiment, road elements in the first road image can be clustered in advance to determine multiple clustered groups in the first road image, and road elements in the second road image can be clustered in advance to determine multiple clustered groups in the second road image.

[0073] In the second road image, the cluster group corresponding to the first target cluster group refers to the cluster group in the second road image where the feature points associated with the feature points in the first target cluster group are located.

[0074] In this embodiment, after clustering the road elements, the transition matrix is ​​determined based on the feature pairs in the first cluster group to which the first road element in the first road image belongs and the corresponding second cluster group in the second road image. This can effectively reduce the impact of mistaking noise points or outliers as feature points on the transition matrix, thus achieving a good smoothing effect and ensuring the accuracy and effectiveness of subsequent semantic association based on the transition matrix.

[0075] In some embodiments, before step 210, the method further includes: clustering the road elements in the first road image according to the depth estimation information corresponding to the first road image and the position information of each road element in the first road image to determine the clustered group in the first road image; clustering the road elements in the second road image according to the depth estimation information corresponding to the second road image and the position information of each road element in the second road image to determine the clustered group in the second road image; and determining the corresponding clustered group in the first road image and the second road image according to the feature point association result between the first road image and the second road image.

[0076] The depth estimation information corresponding to the first road image is used to indicate the depth of each pixel in the first road image. The depth of the pixel referred to in the depth estimation information can be a relative depth.

[0077] In a specific embodiment, a depth prediction network model can be used to estimate the depth of each pixel in the first road image and output depth estimation information corresponding to the first road image. This depth estimation information can be a relative depth map corresponding to the first road image. The depth prediction network model can be constructed using one or more neural network models. To ensure the accuracy of the depth estimation information output by the depth prediction network model, it needs to be trained first using second training data.

[0078] The second training data includes multiple second training images and a first relative depth map corresponding to each second training image, wherein the first relative depth map is used to indicate the relative depth of each pixel in the second training image.

[0079] The training process is as follows: The second training image is input into the depth prediction network model, which performs relative depth prediction on the second training image and outputs the predicted relative depth map corresponding to the second training image; then, the function value of the second loss function corresponding to the depth prediction network model is calculated based on the predicted relative depth map and the corresponding first relative depth map, and the parameters of the depth prediction network model are adjusted in reverse according to the function value until the second training termination condition is met.

[0080] The process of clustering road elements is explained below using the first road image as an example: The positional information of each road element in the first road image reflects its location within the first road image. The depth estimation information corresponding to the first road image refers to the depth information of the pixels in each road element within the first road image. Based on this, multiple road elements in the first road image whose distance difference does not exceed a fifth threshold and whose depth difference does not exceed a sixth threshold can be grouped together, forming a single cluster. Clustering based on the positional information and corresponding depth estimation information of road elements ensures that multiple road elements in a single cluster are located in relatively close-to-each-other regions within the first road image.

[0081] It is understandable that a cluster in the first (or second) road image contains one or more road elements. The process of clustering road elements in the second road image is similar to that described above and will not be repeated here.

[0082] Figure 4 An exemplary diagram illustrates the clustering of multiple road elements in a road image. Figure 4The image shows a depth map corresponding to a road image. The bounding boxes in this depth map are the detection boxes corresponding to road elements in the road image. The numbers in the detection boxes represent the depth of the road element. Figure 4 As shown, the bounding boxes corresponding to the three road elements in the upper left corner of the depth map are close to each other, and the depth differences between the three road elements are small. Therefore, these three road elements can be grouped into one aggregation group. And... Figure 4 The road element in the lower left corner is far from the detection box corresponding to the three road elements. Therefore, the road element in the lower left corner and the three road elements in the upper right corner belong to two different aggregation groups.

[0083] In a specific embodiment, the location information of a road element in a road image (e.g., a first road image, a second road image) can be indicated by a detection box surrounding the road element. In this case, clustering road elements can also be understood as clustering detection boxes. A cluster can include one or more detection boxes corresponding to road elements.

[0084] In a specific embodiment, a rectangular bounding box can be used to indicate the location information of an aggregation group in a road image. In this case, the bounding box is used to enclose the detection boxes of road elements belonging to an aggregation group. For example, in Figure 4 In the image, the location of the aggregate group in the road image is indicated by a dashed rectangle.

[0085] As described above, the transition matrix corresponding to the road element needs to be calculated using the position information of a certain number of corresponding feature point pairs. In this case, in order to ensure that the transition matrix can be calculated accurately in the future, it is necessary to ensure that the number of feature points in an aggregation group is not less than a certain threshold.

[0086] In some embodiments, the method further includes: if there is an aggregate group in the first road image or the second road image with a number of feature points less than a first threshold, then the aggregate group is expanded so that the number of feature points in the expanded aggregate group is not less than the first threshold.

[0087] In other words, after clustering road elements in a road image, the number of feature points in a cluster is counted. If the number of feature points in a cluster is less than a first threshold, the cluster is dilated. After dilation, the number of feature points in the cluster is not less than the first threshold. The first threshold can be set according to actual needs, for example, 10, 15, 20, etc.

[0088] Expanding an aggregate group refers to widening the area covered by the aggregate group, thereby increasing the number of feature points falling within it. In a specific embodiment, the rectangle representing the boundary of the aggregate group can be enlarged proportionally to achieve the expansion of the aggregate group.

[0089] Figure 5 This is a schematic diagram illustrating the expansion of a polymer group according to an embodiment of this application. Figure 5 The left image in the diagram is a schematic diagram of the feature points in a cluster before dilation, as shown below. Figure 5 As shown, the number of feature points within the rectangle representing the boundary of the aggregation group is less than the first threshold. This aggregation group can then be expanded. Figure 5 The right figure in the diagram shows a schematic diagram of the polymer group after expansion, as shown below. Figure 5 As shown, after expansion, the area enclosed by the rectangle corresponding to the aggregation group is wider, and correspondingly, the number of feature points falling into the aggregation group increases, satisfying the condition of not less than the first threshold.

[0090] In this embodiment, when the number of feature points in an aggregate group is small, the aggregate group is expanded so that the number of feature points in the expanded aggregate group is not less than the first threshold, thereby ensuring that the transition matrix is ​​calculated based on the feature points in the aggregate group.

[0091] Step 220: Based on the transition matrix corresponding to the road element that matches the feature point, semantically associate the road element in the first road image with the road element in the second road image, wherein the two road elements that are semantically associated represent the same road element in space.

[0092] Semantic association refers to associating road elements representing the same road element in two different road images. It can be understood that the two road elements semantically associated are road elements located in the two different road images.

[0093] In some embodiments, such as Figure 6 As shown, step 220 includes:

[0094] Step 610: For the first road element in the first road image, determine the prediction box corresponding to the first detection box in the second road image based on the position information of the first detection box corresponding to the first road element and the transition matrix corresponding to the first road element.

[0095] Step 620: If there is a second detection box corresponding to a second road element of the target element type in the prediction box, then the first road element in the first road image and the second road element in the second road image are semantically associated; wherein, the element type to which the first road element belongs is the target element type.

[0096] The first road element refers to the road element in the first road image to be semantically associated. The detection box used to indicate the position of the first road element in the first road image is called the first detection box. The target element type refers to the element type to which the first road element belongs. The position information of the first detection box corresponding to the first road element and the element type to which the first road element belongs can be obtained from the target detection results corresponding to the first road image.

[0097] The second road element refers to a road element in the second road image that is located within a prediction bounding box and whose element type is the target element type. The second detection box refers to a detection box used to indicate the location of the second road element. Similarly, the position information of the detection boxes corresponding to each road element in the second road image and the element type to which each road element belongs can be obtained from the target detection results corresponding to the second road image.

[0098] As described above, the transition matrix corresponding to the first road element indicates the transformation relationship between the first road element in the first road image and the road elements in the second road image that match the feature points of the first road element. Step 610 is equivalent to performing a reverse calculation based on the transition matrix corresponding to the first road element; that is, transforming the first detection box in the first road image to the second road image according to the transformation relationship indicated by the transition matrix. The detection box obtained by transforming the first detection box to the second road image is called the prediction box corresponding to the first detection box in the second road image.

[0099] In some embodiments, multiple reference points (e.g., the four endpoints of the first detection box) can be selected from the first detection box, and then transformed according to the transformation matrix to obtain the associated points corresponding to each reference point. The determined associated points are located on the prediction box corresponding to the first detection box, thereby determining the position of the prediction box in the second road image.

[0100] If a road element exists within the prediction bounding box of the second road image, and the element type of the road element is the same as that of the first road element, then the road element can be determined to be the same road element in the representation space of the first road element in the first road image. Thus, the road element in the second road image can be semantically associated with the first road element in the first road image.

[0101] Figure 6 A schematic diagram of the first detection box 611 corresponding to the road element representing a speed limit sign in the first road image is shown. By transforming the first detection box 611 according to the transition matrix corresponding to the speed limit sign, the following can be obtained: Figure 6 Predicted bounding box 612 in the second road image. (By...) Figure 6As can be seen from B in the image, there is a detection box within the prediction box 612, and the road element enclosed by the detection box is also a speed limit sign. Therefore, it can be determined that the speed limit sign enclosed by the first detection box in the first road image and the speed limit sign in the prediction box 612 in the second road image represent the same speed limit sign in the space.

[0102] In other embodiments of this application, such as Figure 7 As shown, step 220 includes:

[0103] Step 610: For the first road element in the first road image, determine the prediction box corresponding to the first detection box in the second road image based on the position information of the first detection box corresponding to the first road element and the transition matrix corresponding to the first road element.

[0104] Step 640: Dilate the predicted bounding box in the second road image.

[0105] Step 650: If there is a second detection box corresponding to a second road element of the target element type in the predicted box after dilation, then semantically associate the first road element in the first road image with the second road element in the second road image.

[0106] Dilation refers to increasing the area enclosed by the prediction box, or increasing the size of the prediction box, to make the area enclosed by the prediction box wider. In a specific embodiment, the dilation ratio corresponding to the width of the prediction box and the dilation ratio corresponding to the height can be preset, so that the prediction box is dilated according to the corresponding dilation ratio. The dilation ratio is used to indicate the ratio of the length (or height) after dilation to the length (or height) before dilation.

[0107] like Figure 7 Figure A shows a schematic diagram of the first detection box 611 corresponding to the road element representing the speed limit sign in the first road image. By transforming the first detection box 611 according to the transition matrix corresponding to the speed limit sign, we can obtain... Figure 7 The predicted bounding box 612 in the second road image shown in Figure B; then, the predicted bounding box in the second road image is dilated to obtain... Figure 7 The third detection box 613, shown as C in the diagram, is the detection box after the prediction box has been dilated. Comparing the prediction box 612 and the third detection box 613, it can be found that the area enclosed by the third detection box is larger than the area enclosed by the prediction box 612.

[0108] In this embodiment, after the first detection box is transformed into the prediction box in the second road image, the prediction box is first dilated. Then, based on the position information of the dilated prediction box, the road element in the second road image that is semantically associated with the first road element in the first road image is determined, which improves the fault tolerance and robustness of the semantic association of road objects.

[0109] In some embodiments, if it is determined that there is no detection box corresponding to a road element of the target element type within the prediction box in the second road image, then the prediction box in the second road image can be dilated. This avoids dilating the prediction box even when a second detection box exists in the prediction box of the second road image, thus avoiding wasting processing resources.

[0110] Please continue reading. Figure 2 Step 230: Based on the position information of multiple feature points corresponding to the semantically associated road elements in the first road image and the position information in the second road image, determine the three-dimensional spatial information corresponding to the semantically associated road elements.

[0111] Multiple associated feature points between semantically related road elements can also be referred to as feature point pairs corresponding to the semantically related road elements. For ease of description, assuming that the first road element in the first road image and the second road element in the second road image are semantically related road elements, then the feature point in the first road element in the first road image within the feature point pair corresponding to the semantically related road elements is called the first feature point, and the feature point in the second road element in the second road image within the feature point pair corresponding to the semantically related road elements and corresponding to the first feature point is called the second feature point.

[0112] The three-dimensional spatial information corresponding to the semantically associated road element is used to indicate the position of the road element in the actual space. The three-dimensional spatial information may include the three-dimensional coordinates of multiple points of the semantically associated road element in the actual space.

[0113] In step 230, the road element is reconstructed in three dimensions based on images of the same road element from two different perspectives in the actual space. In a specific embodiment, the three-dimensional coordinates of multiple points of the road element in the actual space can be determined based on the position information of multiple feature point pairs of the road element in the first road image and the second road image.

[0114] In some embodiments, the three-dimensional coordinates of each point in the road element in the actual space, i.e., the coordinates in the world coordinate system, can be determined by using a triangulation algorithm based on the position information of multiple feature points corresponding to the semantically associated road element in the first road image and the position information of the second road image.

[0115] Specifically, the relative pose information of the first image acquisition device corresponding to the first road image and the second image corresponding to the second road image can be determined by using the position information of multiple feature points corresponding to semantically associated road elements in the first road image and the position information of the second road image. This relative pose information includes rotation and translation matrices. Then, based on the relative pose information and the intrinsic parameter matrices of the first and second image acquisition devices, the projection matrix of the first road image in the camera coordinate system corresponding to the first image acquisition device and the projection matrix of the second road image in the camera coordinate system corresponding to the second image acquisition device are determined. Finally, based on the projection matrix in the corresponding camera coordinate system and the position information of each pair of feature points in the semantically associated road elements in the first and second road images, the three-dimensional coordinates of the points represented by each pair of feature points in the actual space are calculated.

[0116] In some other embodiments, step 230 includes: determining relative pose information based on the position information of multiple feature points corresponding to the semantically associated road element in the first road image and the position information in the second road image; determining, based on the relative pose information, the first relative position information of the semantically associated road element in the first camera coordinate system corresponding to the first road image and the second relative position information of the semantically associated road element in the second camera coordinate system corresponding to the second road image; and determining the three-dimensional spatial information corresponding to the semantically associated road element based on the interval distance between the shooting position corresponding to the first road image and the shooting position corresponding to the second road image, the first relative position information, and the second relative position information.

[0117] The relative pose information describes the rotation and translation of the first image acquisition device corresponding to the first road image and the second image acquisition device from which the second road image originates. Correspondingly, the relative pose information includes rotation and translation matrices. This relative pose information can also be understood as the transformation relationship between the first camera coordinate system corresponding to the first road image and the second camera coordinate system corresponding to the second road image.

[0118] The first relative position information can be used to indicate the position information of the semantically associated road element in the first camera coordinate system corresponding to the first road image. The second relative position information is used to indicate the position information of the semantically associated road element in the second camera coordinate system corresponding to the second road image.

[0119] As described above, based on relative pose information, the first projection matrix of the first road image in the first camera coordinate system and the second projection matrix of the second road image in the second camera coordinate system can be determined. Therefore, by combining the position information (i.e., pixel coordinates) of the semantically associated road element in the first road image and the first projection matrix, the position information of the road element in the first camera coordinate system can be determined in reverse; similarly, by combining the position information (i.e., pixel coordinates) of the semantically associated road element in the second road image and the second projection matrix, the position information of the semantically associated road element in the second camera coordinate system can be determined in reverse.

[0120] Based on this, according to the first relative position information, the second relative position information, and the distance between the shooting position corresponding to the first road image and the shooting position corresponding to the second road image (or also called the acquisition point), the absolute depth corresponding to the road element in the actual space can be determined according to the principle of pinhole imaging. The following is an explanation of the distance between the road element and the first image acquisition device corresponding to the first road image.

[0121] Figure 8 An exemplary schematic diagram of pinhole imaging is shown below. Please refer to [link / reference]. Figure 8 The absolute depth of each point in the road element can be calculated using the following formula:

[0122]

[0123] Combining the two formulas above, we can obtain:

[0124]

[0125] Where d represents the distance from a point K in the road element in actual space to the first image acquisition device (i.e., C) corresponding to the first road image, which is equivalent to the absolute depth of point K relative to the first image acquisition device. d1 is the interval distance between the shooting position corresponding to the first road image and the shooting position corresponding to the second road image. x1 is the distance on the horizontal axis from the origin of the first camera coordinate system to the pixel point K1 corresponding to point K in the first road image; x2 is the distance on the horizontal axis from the origin of the second camera coordinate system to the pixel point K corresponding to point K in the second road image. X represents the horizontal coordinate of point K in the world coordinate system.

[0126] In some embodiments, prior to step 230, the method further includes: determining a first projection position of a first acquisition point associated with a first road image on the road trajectory corresponding to the first road image, and using the first projection position as the shooting position corresponding to the first road image; determining a second projection position of a second positioning point associated with a second road image on the road trajectory corresponding to the second road image, and using the second projection position as the shooting position corresponding to the second road image; and determining the interval distance between the shooting position corresponding to the first road image and the shooting position corresponding to the second road image based on the first projection position and the second projection position.

[0127] As described above, road images can be collected during the vehicle's journey along a road trajectory. In a specific embodiment, multiple road images collected during the journey along a road trajectory can be stored in a road image set, thus one road image set corresponds to one road trajectory. In this case, the road trajectory corresponding to the first road image refers to the road trajectory corresponding to the road image set from which the first road image originates.

[0128] During the image acquisition process along the road, the location information of the acquisition points is associated with and stored in the acquired road images. In this application, the acquisition point associated with the first road image is referred to as the first acquisition point, and the acquisition point associated with the second road image is referred to as the second acquisition point. The first projection position refers to the projection position of the first acquisition point on the road trajectory corresponding to the first road image.

[0129] Similarly, the road trajectory corresponding to the second road image refers to the road trajectory corresponding to the set of road images from which the second road image originates. The second projection position refers to the projection position of the second acquisition point on the road trajectory corresponding to the second road image.

[0130] Figure 9 An exemplary diagram illustrates the projection from the first acquisition point onto the road trajectory corresponding to the first road image. By projecting the first acquisition point onto the road trajectory corresponding to the first road image, a projection point can be determined. The location of the projection point is the first projection position. It can be understood that the line connecting the projection point and the first acquisition point is perpendicular to the tangent line on the road trajectory passing through the projection point.

[0131] In the above embodiments, by using the projection position as the shooting position of the corresponding road image, the inaccuracy of the three-dimensional spatial information of the road elements due to the inaccuracy of the position information of the stimulus points associated with the road image can be effectively avoided, thus realizing the correction of the position information of the acquisition point of the road image.

[0132] Step 240: Construct a 3D map based on the 3D spatial information of the semantically associated road elements and the element type to which the semantically associated road elements belong.

[0133] Through the above process, the three-dimensional spatial information of multiple points in each road element can be determined. Based on the three-dimensional spatial information of multiple road elements, a model of each road element can be constructed, and a corresponding three-dimensional map of the road can be built.

[0134] In a specific embodiment, for complex roads such as intersections, for example... Figure 10 The intersection shown is divided into four road segments 1010 and an intersection 1020 connecting the road segments 1010. 3D maps are constructed for each road segment 1010 and intersection 1020. Then, the 3D maps corresponding to the four road segments 1010 and the 3D map corresponding to intersection 1020 are stitched together according to the relative positions of the road segments and intersections to obtain the 3D map of the intersection. In a specific embodiment, this can be constructed using five mapping units. Figure 10 The 3D map corresponding to the crossroads shown is provided, in which one road segment corresponds to one mapping unit and one intersection corresponds to one mapping unit.

[0135] In this application, after feature point matching of the first and second road images, a transition matrix is ​​calculated between road elements in the first and second road images whose feature points match. The road elements in the different road images are then semantically associated based on the transition matrix. Subsequently, the 3D spatial information of the road elements is determined based on their positional information in the two road images, and a 3D map is constructed using this information. This process utilizes road images acquired from two different perspectives to construct a 3D map without relying on LiDAR for point cloud data acquisition. This reduces the dependence on LiDAR equipment for 3D map construction and lowers the cost of 3D map construction. Furthermore, this solution allows for timely updates of the 3D map based on the updated road images when roads are updated.

[0136] Furthermore, in the scheme of this application, based on the feature point extraction of two road images, not only are feature points in different road images matched, but also the road elements in the two road images are further semantically associated based on the transition matrix. Compared with the method of associating road elements only based on feature point matching, this scheme references more information, thereby ensuring the accuracy of the semantic association of road elements, thus ensuring the accuracy of the three-dimensional spatial information of the determined road elements, and thus ensuring the accuracy of the generated three-dimensional map.

[0137] In some embodiments of this application, such as Figure 11 As shown, the method also includes:

[0138] Step 1110: If there is a third road element in the first road image that has not been semantically associated, determine the global feature information of the third road element in the first road image based on the local feature information of the third road element in the first road image and the orientation information of at least one fourth road element that has been semantically associated in the first road image relative to the third road element.

[0139] In this application, road elements in the first road image that are not semantically associated are referred to as third road elements. Road elements in the first road image that have been semantically associated are referred to as fourth road elements.

[0140] The local feature information of the third road element in the first road image may include the location information of the third road element in the first road image, the element type to which the third road element belongs, and the feature descriptors of each feature point in the third road element, etc., without specific limitations.

[0141] The orientation information of the fourth road element relative to the third road element is used to indicate the relative distance and relative direction of the fourth road element relative to the third road element in the first road image.

[0142] In some embodiments, the local feature information of the third road element in the first road image and the orientation information of each fourth road element relative to the third road element can be fused to obtain the global feature information of the third road element in the first road image.

[0143] In other embodiments, the global feature information of the third road element in the first road image may also include the element type to which each fourth road element belongs.

[0144] In some embodiments, since road elements that are farther away from the third road element in the first road image have a lower correlation with the third road element and a weaker feature representation of the third road element, road elements that are within the range of the distance from the third road element in the first road image that are less than the second distance threshold and have been semantically associated are used as the fourth road element, while road elements that are farther away from the third road element in the first road image than the second distance threshold are not included in the process of constructing the global feature information of the third road element.

[0145] For example Figure 12In the first road image, the electronic eye is a road image that has not been semantically associated, and the electronic eye 1210 is regarded as the third road element. The first road sign 1211 and the first directional sign 1212 in the first road image are both road elements that have been semantically associated. In this case, the first road sign 1211 and the first directional sign 1212 can be regarded as the fourth road element, and the relative orientation information of the first road sign 1211 and the first directional sign 1212 relative to the electronic eye 1210 (see Table 1 for details) can be used as the contextual features of the electronic eye 1210, thereby determining the global feature information of the electronic eye 1210. In Table 1, the relative distance is equal to the ratio of pixel distance to image height.

[0146] Table 1

[0147] 0 First road sign Bottom right corner (140 degrees) 0.25 1 First signpost Bottom right corner (85 degrees) 0.15

[0148] Step 1120: Based on the local feature information of the fifth road element in the candidate road image that has not been semantically associated and the orientation information of the sixth road element in the candidate road image relative to the fifth road element that has been semantically associated, determine the global feature information of the fifth road element in the candidate road image; the element type to which the fifth road element belongs is the same as the element type to which the third road element belongs.

[0149] Candidate road images can be second road images or other road images containing road elements that are not semantically related. In some embodiments, road images containing road elements that are not semantically related can be obtained from the image group containing the first road image as candidate road images. Specifically, the number of matching pixels between different road images belonging to the same image group exceeds a second threshold. Furthermore, the distance between the acquisition points corresponding to different road images belonging to the same image group does not exceed a third threshold, and the corresponding acquisition orientation angle interpolation does not exceed a fourth threshold. This ensures that the selected candidate road images present essentially the same road scene as the first road image, thus avoiding invalid similarity calculations.

[0150] The fifth road element refers to a road element in the candidate road image that has not been semantically associated. The sixth road element refers to a road element in the candidate road image that has been semantically associated.

[0151] The determination of the global feature information of the fifth road element in the candidate road image is similar to the determination of the global feature information of the third road element, and will not be repeated here.

[0152] Step 1130: Calculate the similarity between the third road element and the fifth road element based on the global feature information of the third road element in the first road image and the global feature information of the fifth road element in the candidate road image.

[0153] Specifically, the similarity between the global feature information of the third road element in the first road image and the global feature information of the fifth road element in the candidate road image can be calculated, and the calculated similarity can be used as the similarity between the third road element and the fifth road element.

[0154] Step 1140: If the similarity exceeds the similarity threshold, then semantically associate the third road element in the first road image with the fifth road element in the candidate road image.

[0155] Conversely, if the similarity does not exceed the similarity threshold, then the third road element in the first road image and the fifth road element in the candidate road image are determined to represent different road elements in the actual space.

[0156] It is understandable that by semantically associating the third road element in the first road image with the fifth road element in the candidate road image, the three-dimensional spatial information of the corresponding road element can be determined, and then used to construct a three-dimensional map.

[0157] In practice, road images vary considerably due to factors such as shooting angle and lighting. Therefore, the semantic association of road elements based on the transition matrix may not be comprehensive. In such cases, one can follow... Figure 13 In a corresponding implementation, a fuzzy matching extension method is used to extend the semantic association to more road elements, ensuring the comprehensiveness of the semantic association.

[0158] In some embodiments, the first road image and the second road image belong to the same image group; the method further includes: filtering out feature points located in the first interference pixel region in the first road image according to the location information of the first interference pixel region indicated by the semantic segmentation result of the first road image; filtering out feature points located in the second interference pixel region in the second road image according to the location information of the second interference pixel region indicated by the semantic segmentation result of the second road image; associating the feature points in the filtered first road image with the feature points in the filtered second road image; if the number of associated feature point pairs exceeds a second threshold, then determining that the first road image and the second road image belong to the same image group.

[0159] In this application, the interfering pixel regions in the road image include pixel regions corresponding to non-road elements in the road image, as well as pixel regions that interfere with relative distance estimation (i.e., calculating the relative distance between road elements involved in the clustering process of road elements mentioned above). Specifically, in this application, the interfering pixel regions include the pixel region where the vehicle is located, the pixel region where the adjacent vehicle is located, the pixel region where the pedestrian is located, and the watermark in the road image.

[0160] In this application, for ease of distinction, the interfering pixel region in the first road image is referred to as the first interfering pixel region, and the interfering pixel region in the second road image is referred to as the second interfering pixel region.

[0161] In some embodiments, a semantic segmentation network model can be used to perform semantic segmentation on road images (a first road image and a second road image) to obtain corresponding semantic segmentation results. These semantic segmentation results are used to indicate the location of interfering pixel regions in the road image. The semantic segmentation network model can be a model constructed using one or more neural networks.

[0162] To ensure the accuracy of semantic segmentation, the semantic segmentation network model can be trained using third training data. This third training data includes multiple third training images and corresponding label information for each image. The label information indicates the location of interfering pixel regions within the third training images. During training, the third training images are input into the semantic segmentation network model, which performs semantic segmentation and outputs a predicted semantic segmentation result. This predicted result indicates the location of the predicted interfering pixel regions within the third training images.

[0163] Then, based on the label information corresponding to the third training image and the predicted semantic segmentation result corresponding to the third training image, the function value of the third loss function corresponding to the semantic segmentation network model is calculated, and the parameters of the semantic segmentation network model are adjusted in reverse according to the function value until the third training termination condition is reached.

[0164] Figure 13 This is a schematic diagram illustrating the filtering of feature points in a road image according to an embodiment of this application. Figure 13 Image A is a schematic diagram of feature points in the road image before filtering. Figure 13 Figure B in the image is a schematic diagram of feature points in the filtered road image. After filtering out feature points in the interference pixel regions, the corresponding feature points in the interference pixel regions do not participate in subsequent calculation processes (such as the calculation of the transition matrix).

[0165] In this embodiment, feature points in the interfering pixel regions of the two road images are first filtered out, and then feature points in the filtered two road images are associated. Since the subsequent semantic association of road elements basically does not involve the use of feature points in the interfering pixel regions, the amount of data processing can be effectively reduced and the waste of processing resources can be avoided.

[0166] Understandably, the more feature points associated in the two filtered road images, the higher the probability that the two road images were captured from the same road environment, and the more identical road elements are represented in the two road images. Thus, the two road images are grouped into the same image group. Then, in step 210, two road images can be obtained from the same image group to perform 3D reconstruction of the identical road elements, thereby improving the efficiency of 3D reconstruction.

[0167] In this embodiment, feature points are extracted from the two road images and then the feature points in the interference pixel region are filtered out. The filtered two road images are then associated with feature points, instead of removing the interference pixel region from the two road images before extracting and associating feature points. This avoids mistaking points on the edge of the interference pixel region as feature points and ensures the accuracy of the feature points involved in the subsequent feature point association.

[0168] In some embodiments, the first road image and the second road image are obtained from an initial image set; in this embodiment, the method further includes:

[0169] For multiple road images to be processed, the multiple road images are grouped according to the acquisition location information and acquisition orientation angle of each road image to obtain at least one initial image group; wherein, the interval between acquisition points of any two road images in the same initial image group does not exceed a distance threshold and the difference between acquisition orientation angles of any two road images does not exceed a third threshold.

[0170] In some embodiments, the interval distance between corresponding acquisition points can be calculated based on the location information of the acquisition points associated with the two road images. In other embodiments, if according to Figure 9 If the shooting position corresponding to the road image is corrected, the interval distance between the corresponding acquisition points is calculated according to the corrected shooting position.

[0171] It is understandable that if the distance between two road images does not exceed the distance threshold and the difference in the corresponding acquisition orientation angle does not exceed the third threshold, it indicates that the road scenes presented by the two road images are basically the same, and thus, it indicates that the probability of the two road images having the same road elements is higher.

[0172] After determining the initial image group, two road images can be selected from the initial images to perform feature point filtering and feature point association according to the above process. This avoids performing feature point filtering and feature point association on two road images that are significantly different, thus avoiding wasting processing resources and ensuring the effectiveness of the feature point filtering and feature point association.

[0173] The solution of this application will now be described in detail with reference to a specific embodiment.

[0174] Figure 14 This is a flowchart illustrating a method for constructing a 3D map according to a specific embodiment of this application. This method can be executed by a server, such as... Figure 14 As shown, it specifically includes:

[0175] Step 1410, position correction.

[0176] Specifically, an in-vehicle monocular camera (such as a dashcam) can be used to capture images of the road environment ahead of the vehicle. This can be achieved by capturing images at fixed time intervals or uniform distance intervals. In practice, the location information of the captured points associated with the road images may be inaccurate. Therefore, the location information of the captured points is corrected based on a road trajectory map. The specific correction process can be found above. Figure 9 The corresponding process will not be elaborated here.

[0177] Subsequently, based on the location information of the acquisition points corresponding to the road images, multiple road images with an interval distance between the corresponding acquisition points not exceeding the third threshold and an angle difference between the corresponding acquisition directions not exceeding the fourth threshold can be grouped into the same initial image group. Then, two road images can be selected from the same initial image group for target association to reduce the amount of computation.

[0178] Step 1420: Image content parsing.

[0179] In this embodiment, a multi-task parsing network is used to perform content parsing on each road image. The content parsing performed on the road images in this embodiment includes: object detection, semantic analysis, and depth estimation.

[0180] Figure 15 This is a schematic diagram of the structure of a multi-task parsing network according to an embodiment of this application. Figure 15 As shown, the multi-task parsing network consists of a common backbone network and three different head sub-neural networks (object detection head network, semantic segmentation head network, and depth estimation head network). The backbone network is used for common feature extraction of road images.

[0181] In a specific embodiment, the target detection head network can be a network structure based on the YOLOv3 detection head. The target detection head network is used to detect road elements included in the road image, such as road signs, warning signs, danger signs, etc., and the location information of the road elements in the road image (such as the position of the detection box) and the element type to which the road element belongs are input accordingly.

[0182] The semantic segmentation head network can be a semantic segmentation network head similar to DeepLabV3. This semantic segmentation head network can identify the interfering pixel regions in the road image that interfere with the relative distance estimation. The interfering pixel regions can be the pixel region where the vehicle is located, the pixel region where the adjacent vehicle is located, the pixel region where the pedestrian is located, and the watermark in the road image.

[0183] A depth estimation network can be a model trained using depth information labeled on the KITTI standard dataset, which can then be used to predict relative depth.

[0184] Understandable, Figure 15 The backbone network + object detection head network is equivalent to the object detection model mentioned above; the backbone network + semantic segmentation head network is equivalent to the semantic segmentation network model mentioned above; the backbone network + depth estimation network is equivalent to the depth prediction network model mentioned above.

[0185] Step 1430, Target Association. Specifically, step 1430 includes steps 1431-1434, where step 1431 is feature point extraction. In a specific embodiment, image feature points can be extracted using a SuperPoint network (an end-to-end feature point and descriptor extraction network).

[0186] Step 1432, Feature Point Matching. Specifically, this step includes two steps: interference region removal and full feature point matching. In the interference region removal step, feature points located in interference pixel regions in the road image are filtered out. Specifically, interference pixel regions can be segmented based on the semantic segmentation results output by the semantic segmentation head network, and feature points falling within the interference pixel regions can be filtered out based on their coordinates.

[0187] In the full feature point matching step, feature points in the filtered two road images are associated, linking the most similar feature points between the two road images. Specifically, the filtered feature points can be matched using the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm or the SuperGlue algorithm (a feature matching algorithm based on graph convolutional neural networks) to obtain the matching relationship of the entire two road images. In a specific embodiment, if there are more than M matching feature points in the entire image, the two road images are considered to be matched, that is, the two road images are grouped into one image group. M can be selected as needed, for example, M is 40, 45, 50, 100, etc.

[0188] Step 1433: Calculate the local transfer matrix. The process of calculating the local transfer matrix is ​​the same as the process described in step 120 above, which calculates the transfer matrix between road elements in the first road image and feature points in the second road image that match each other. See the description above for details, which will not be repeated here.

[0189] Step 1434: Perform target association based on the transition matrix. This step is equivalent to step 220 above, as detailed above, and will not be repeated here.

[0190] Step 1440, Fuzzy Matching Extension.

[0191] Specifically, step 1440 includes the processes of generating contextual features and feature matching. Generating contextual features corresponds to steps 1110 and 1120 above, and the feature matching process corresponds to step 1130 above. The specific implementation process is described above and will not be repeated here.

[0192] Step 1450, Change Discovery.

[0193] Change detection can be performed on road images belonging to the same image group based on the order of their acquisition time. Specifically, if N road images are acquired near a location point, and these N roads belong to the same image group, and if a certain road element is found in all of the 1st to (n-1th)th road images according to the acquisition time order, while a road element that is not semantically associated exists in the nth road image, then this unrelated road element can be considered a newly added road element and marked. In subsequent processes, road images including this newly added road element can be acquired, and its 3D spatial information can be specifically determined to add it to the 3D map, thus updating the 3D map.

[0194] Step 1460: Construct a graph based on the semantic association results. This step is equivalent to steps 230-240 above. For specific implementation details, please refer to the description above, which will not be repeated here.

[0195] Based on the solution in this embodiment, a road scene can be modeled in three dimensions by acquiring multiple images of the road environment, and a corresponding three-dimensional map can be constructed. This solution can be used as a basic technology in application scenarios such as road production automation, high-precision map automation, and autonomous driving.

[0196] In this embodiment, information collection is not required through devices such as LiDAR. Instead, a 3D map is constructed using images collected multiple times for the road environment, which reduces the cost of 3D map construction. Moreover, road elements in the road can be updated in a timely manner.

[0197] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0198] Figure 16 This is a block diagram illustrating an apparatus for constructing a 3D map according to an embodiment of this application. This apparatus can be configured in a server to implement the method for constructing a 3D map provided in this application. Figure 16 As shown, the apparatus for constructing a 3D map includes: a transition matrix determination module 1610, used to calculate the transition matrix between road elements in a first road image and feature points in a second road image; the first road image and the second road image are different road images collected from two acquisition points with an interval distance not exceeding a distance threshold; a semantic association module 1620, used to semantically associate road elements in the first road image and road elements in the second road image according to the transition matrix corresponding to the road elements whose feature points match, wherein the two semantically associated road elements represent the same road element in space; a 3D spatial information determination module 1630, used to determine the 3D spatial information corresponding to the semantically associated road element according to the position information of multiple feature points corresponding to the semantically associated road element in the first road image and the position information in the second road image; and a map construction module 1640, used to construct a 3D map according to the 3D spatial information of the semantically associated road element and the element type to which the semantically associated road element belongs.

[0199] In some embodiments, the semantic association module 1620 includes: a prediction box determination unit, configured to determine, for a first road element in a first road image, a prediction box corresponding to the first detection box in a second road image based on the position information of the first detection box corresponding to the first road element and the transition matrix corresponding to the first road element; and a first semantic association unit, configured to semantically associate the first road element in the first road image with the second road element in the second road image if there is a second detection box corresponding to a second road element of the target element type within the prediction box; wherein the element type to which the first road element belongs is the target element type.

[0200] In other embodiments, the semantic association module 1620 includes: a prediction box determination unit, configured to determine, for a first road element in a first road image, a prediction box corresponding to the first detection box in a second road image based on the position information of the first detection box corresponding to the first road element and the transition matrix corresponding to the first road element; a first dilation processing unit, configured to perform dilation processing on the prediction box in the second road image; and a second semantic association unit, configured to semantically associate the first road element in the first road image with the second road element in the second road image if there is a second detection box corresponding to a second road element of the target element type within the dilated prediction box.

[0201] In some embodiments, the apparatus for constructing a 3D map further includes: a first global feature information determination module, configured to determine global feature information of a third road element in a first road image based on local feature information of the third road element in the first road image and the orientation information of at least one fourth road element in the first road image that has been semantically associated with the third road element relative to the third road element if there is a third road element in the first road image that has not been semantically associated with the third road element; a second global feature information determination module, configured to determine global feature information of a fifth road element in a candidate road image based on local feature information of a fifth road element that has not been semantically associated with the fifth road element in a candidate road image and the orientation information of a sixth road element in the candidate road image that has been semantically associated with the fifth road element relative to the fifth road element; wherein the element type to which the fifth road element belongs is the same as the element type to which the third road element belongs; a similarity calculation module, configured to calculate the similarity between the third road element and the fifth road element based on the global feature information of the third road element in the first road image and the global feature information of the fifth road element in the candidate road image; and a third semantic association module, configured to semantically associate the third road element in the first road image with the fifth road element in the candidate road image if the similarity exceeds a similarity threshold.

[0202] In some embodiments, the three-dimensional spatial information determination module 1630 includes: a relative pose information determination unit, configured to determine relative pose information based on the position information of multiple feature points corresponding to the semantically associated road element in a first road image and the position information in a second road image; a relative position information determination unit, configured to determine, based on the relative pose information, a first relative position information of the semantically associated road element in a first camera coordinate system corresponding to the first road image and a second relative position information of the semantically associated road element in a second camera coordinate system corresponding to the second road image; and a three-dimensional spatial information determination unit, configured to determine the three-dimensional spatial information corresponding to the semantically associated road element based on the interval distance between the shooting position corresponding to the first road image and the shooting position corresponding to the second road image, the first relative position information, and the second relative position information.

[0203] In some embodiments, the apparatus for constructing a three-dimensional map further includes: a first shooting position determination module, configured to determine a first projection position of a first acquisition point associated with a first road image on the road trajectory corresponding to the first road image, and use the first projection position as the shooting position corresponding to the first road image; a second shooting position determination module, configured to determine a second projection position of a second positioning point associated with a second road image on the road trajectory corresponding to the second road image, and use the second projection position as the shooting position corresponding to the second road image; and an interval distance determination module, configured to determine the interval distance between the shooting position corresponding to the first road image and the shooting position corresponding to the second road image based on the first projection position and the second projection position.

[0204] In some embodiments, the transition matrix determination module 1610 includes: a feature point pair determination unit, configured to determine multiple feature point pairs corresponding to the road elements whose feature points match based on the road elements whose feature points match in the first road image and the second road image, wherein each feature point pair includes a first feature point in the road elements whose feature points match in the first road image and a second feature point in the road elements whose feature points match in the second road image; and a first transition matrix determination unit, configured to determine the transition matrix corresponding to the road elements whose feature points match based on the position information of the first feature point and the position information of the second feature point in the multiple feature point pairs.

[0205] In other embodiments, the transition matrix determination module 1610 includes: a first target aggregation group determination unit, configured to determine a first target aggregation group to which a first target road element belongs for a first target road element in a first road image to which a transition matrix is ​​to be calculated; a second transition matrix determination unit, configured to determine a transition matrix between the first target aggregation group and the second target aggregation group based on multiple feature point pairs that have a corresponding relationship in the first target aggregation group and the second target aggregation group, wherein the second target aggregation group refers to an aggregation group in the second road image that corresponds to the first target aggregation group; and a third transition matrix determination unit, configured to use the transition matrix between the first target aggregation group and the second target aggregation group as the transition matrix between the first target road element in the first road image and the second target road element in the second road image; wherein the second target road element refers to a road element in the second road image whose feature point is located in the second target aggregation group.

[0206] In some embodiments, the apparatus for constructing a three-dimensional map further includes: a first clustering module, configured to cluster road elements in the first road image according to depth estimation information corresponding to the first road image and position information of each road element in the first road image, to determine an aggregate group in the first road image; a second clustering module, configured to cluster road elements in the second road image according to depth estimation information corresponding to the second road image and position information of each road element in the second road image, to determine an aggregate group in the second road image; and an aggregate group determination module, configured to determine the corresponding aggregate groups in the first road image and the second road image according to the feature point association results between the first road image and the second road image.

[0207] In some embodiments, the apparatus for constructing a three-dimensional map further includes: a second dilation processing module, configured to dilate the aggregate group if there is an aggregate group in the first road image or the second road image with a number of feature points less than a first threshold, so that the number of feature points in the dilated aggregate group is not less than the first threshold.

[0208] In some embodiments, the first road image and the second road image belong to the same image group; the apparatus for constructing a three-dimensional map further includes: a first filtering module, configured to filter out feature points located in the first interference pixel region of the first road image based on the location information of the first interference pixel region indicated by the semantic segmentation result of the first road image; a second filtering module, configured to filter out feature points located in the second interference pixel region of the second road image based on the location information of the second interference pixel region indicated by the semantic segmentation result of the second road image; a feature point association module, configured to associate feature points in the filtered first road image with feature points in the filtered second road image; and an image group determination module, configured to determine that the first road image and the second road image belong to the same image group if the number of associated feature point pairs exceeds a second threshold.

[0209] In some embodiments, the first road image and the second road image are obtained from an initial image group; the apparatus for constructing a three-dimensional map further includes: a grouping module, used to group the multiple road images to be processed according to the acquisition location information and the acquisition orientation angle corresponding to each road image, to obtain at least one initial image group; wherein the interval distance between acquisition points corresponding to any two road images in the same initial image group does not exceed a distance threshold and the difference in acquisition orientation angle corresponding to any two road images does not exceed a third threshold.

[0210] Figure 17 A schematic diagram of a computer system suitable for implementing an electronic device according to embodiments of this application is shown. The electronic device may be... Figure 1The server in this application is an electronic device used to execute the method for constructing a 3D map provided in this application. It should be noted that... Figure 17 The computer system 1700 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0211] like Figure 17 As shown, the computer system 1700 includes a Central Processing Unit (CPU) 1701, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1702 or programs loaded from storage portion 1708 into Random Access Memory (RAM) 1703. The RAM 1703 also stores various programs and data required for system operation. The CPU 1701, ROM 1702, and RAM 1703 are interconnected via a bus 1704. An Input / Output (I / O) interface 1705 is also connected to the bus 1704.

[0212] The following components are connected to I / O interface 1705: an input section 1706 including a keyboard, mouse, etc.; an output section 1707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1708 including a hard disk, etc.; and a communication section 1709 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1709 performs communication processing via a network such as the Internet. Drive 1710 is also connected to I / O interface 1705 as needed. Removable media 1711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1710 as needed so that computer programs read from them can be installed into storage section 1708 as needed.

[0213] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1709, and / or installed from removable medium 1711. When the computer program is executed by central processing unit (CPU) 1701, it performs various functions defined in the system of this application.

[0214] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0215] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0216] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0217] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0218] According to one aspect of this application, an electronic device is also provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement the methods of any of the above embodiments.

[0219] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the methods of any of the above embodiments.

[0220] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0221] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0222] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0223] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for constructing a three-dimensional map, characterized in that, The method includes: Calculating the transition matrix between road elements in a first road image and feature points in a second road image includes: for a first target road element in the first road image to which the transition matrix is ​​to be calculated, determining the first target aggregation group to which the first target road element belongs; determining the transition matrix between the first target aggregation group and the second target aggregation group based on multiple feature point pairs with corresponding relationships in the first and second target aggregation groups, wherein the second target aggregation group refers to the aggregation group in the second road image that corresponds to the first target aggregation group; using the transition matrix between the first target aggregation group and the second target aggregation group as the transition matrix between the first target road element in the first road image and the second target road element in the second road image; the second target road element refers to the road element in the second road image whose feature point is located in the second target aggregation group; the first road image and the second road image are different road images acquired from two acquisition points with an interval distance not exceeding a distance threshold; Based on the transition matrix corresponding to the road element matched by the feature point, the road element in the first road image and the road element in the second road image are semantically associated, wherein the two road elements that are semantically associated represent the same road element in space; Based on the position information of multiple feature points corresponding to the semantically associated road elements in the first road image and the position information in the second road image, the three-dimensional spatial information corresponding to the semantically associated road elements is determined. A 3D map is constructed based on the 3D spatial information of the semantically associated road elements and the element type to which the semantically associated road elements belong.

2. The method according to claim 1, characterized in that, The step of semantically associating road elements in the first road image with road elements in the second road image based on the transition matrix corresponding to the road elements matched by feature points includes: For a first road element in the first road image, the prediction box corresponding to the first detection box in the second road image is determined based on the position information of the first detection box corresponding to the first road element and the transition matrix corresponding to the first road element. If there is a second detection box corresponding to a second road element of the target element type within the prediction box, then the first road element in the first road image and the second road element in the second road image are semantically associated; wherein, the element type to which the first road element belongs is the target element type.

3. The method according to claim 1, characterized in that, The step of semantically associating road elements in the first road image with road elements in the second road image based on the transition matrix corresponding to the road elements matched by feature points includes: For a first road element in the first road image, the prediction box corresponding to the first detection box in the second road image is determined based on the position information of the first detection box corresponding to the first road element and the transition matrix corresponding to the first road element. The prediction box is inflated in the second road image; If there is a second detection box corresponding to a second road element of the target element type within the predicted box after dilation, then the first road element in the first road image and the second road element in the second road image are semantically associated.

4. The method according to claim 2 or 3, characterized in that, The method further includes: If there is a third road element in the first road image that is not semantically associated, the global feature information of the third road element in the first road image is determined based on the local feature information of the third road element in the first road image and the orientation information of at least one fourth road element that is semantically associated in the first road image relative to the third road element. Based on the local feature information of the fifth road element that is not semantically associated in the candidate road image and the orientation information of the sixth road element that is semantically associated in the candidate road image relative to the fifth road element, the global feature information of the fifth road element in the candidate road image is determined; the element type to which the fifth road element belongs is the same as the element type to which the third road element belongs. The similarity between the third road element and the fifth road element is calculated based on the global feature information of the third road element in the first road image and the global feature information of the fifth road element in the candidate road image. If the similarity exceeds the similarity threshold, then the third road element in the first road image is semantically associated with the fifth road element in the candidate road image.

5. The method according to claim 1, characterized in that, The step of determining the three-dimensional spatial information corresponding to the semantically associated road element based on the position information of multiple feature points corresponding to the semantically associated road element in the first road image and the position information of the second road image includes: Relative pose information is determined based on the position information of multiple feature points corresponding to the semantically associated road elements in the first road image and the position information in the second road image; Based on the relative pose information, determine the first relative position information of the semantically associated road element in the first camera coordinate system corresponding to the first road image, and the second relative position information of the semantically associated road element in the second camera coordinate system corresponding to the second road image; Based on the distance between the shooting positions corresponding to the first road image and the second road image, the first relative position information, and the second relative position information, the three-dimensional spatial information corresponding to the semantically associated road element is determined.

6. The method according to claim 5, characterized in that, Before determining the three-dimensional spatial information corresponding to the semantically associated road element based on the interval distance between the shooting positions corresponding to the first road image and the second road image, the first relative position information, and the second relative position information, the method further includes: Determine the first projection position of the first acquisition point associated with the first road image on the road trajectory corresponding to the first road image, and use the first projection position as the shooting position corresponding to the first road image; Determine the second projection position of the second positioning point associated with the second road image on the road trajectory corresponding to the second road image, and use the second projection position as the shooting position corresponding to the second road image; Based on the first projection position and the second projection position, the distance between the shooting position corresponding to the first road image and the shooting position corresponding to the second road image is determined.

7. The method according to claim 1, characterized in that, The calculation of the transition matrix between road elements in the first road image that match feature points in the second road image includes: Based on the road elements whose feature points match in the first road image and the second road image, multiple feature point pairs corresponding to the road elements whose feature points match are determined. Each feature point pair includes a first feature point in the road elements whose feature points match in the first road image and a second feature point in the road elements whose feature points match in the second road image. Based on the position information of the first feature point and the second feature point in the plurality of feature point pairs, the transition matrix corresponding to the road element that matches the feature point is determined.

8. The method according to claim 7, characterized in that, Before calculating the transition matrix between road elements in the first road image that match feature points in the second road image, the method further includes: Based on the depth estimation information corresponding to the first road image and the position information of each road element in the first road image, the road elements in the first road image are clustered to determine the clustered groups in the first road image. Based on the depth estimation information corresponding to the second road image and the position information of each road element in the second road image, the road elements in the second road image are clustered to determine the clustered groups in the second road image; Based on the feature point association results between the first road image and the second road image, corresponding aggregation groups are determined in the first road image and the second road image.

9. The method according to claim 8, characterized in that, The method further includes: If there is an aggregate group in the first road image or the second road image with a number of feature points less than a first threshold, then the aggregate group is expanded so that the number of feature points in the expanded aggregate group is not less than the first threshold.

10. The method according to claim 1, characterized in that, The first road image and the second road image belong to the same image group; The method further includes: Based on the location information of the first interference pixel region indicated by the semantic segmentation result of the first road image, feature points located in the first interference pixel region in the first road image are filtered out. Based on the location information of the second interference pixel region indicated by the semantic segmentation result of the second road image, the feature points located in the second interference pixel region in the second road image are filtered out; The feature points in the filtered first road image are correlated with the feature points in the filtered second road image; If the number of associated feature point pairs exceeds the second threshold, then the first road image and the second road image are determined to belong to the same image group.

11. The method according to claim 10, characterized in that, The first road image and the second road image were obtained from the initial image set; The method further includes: For multiple road images to be processed, the multiple road images are grouped according to the acquisition location information and acquisition orientation angle of each road image to obtain at least one initial image group; wherein, the interval distance between acquisition points of any two road images in the same initial image group does not exceed a distance threshold and the difference in acquisition orientation angle of any two road images does not exceed a third threshold.

12. An apparatus for constructing a three-dimensional map, characterized in that, The device includes: A transition matrix determination module is used to calculate the transition matrix between road elements in a first road image that match feature points in a second road image. The module includes: a first target aggregation group determination unit, used to determine the first target aggregation group to which the first target road element in the first road image to which the transition matrix is ​​to be calculated belongs; a second transition matrix determination unit, used to determine the transition matrix between the first target aggregation group and the second target aggregation group based on multiple corresponding feature point pairs in the first and second target aggregation groups, wherein the second target aggregation group refers to the aggregation group in the second road image corresponding to the first target aggregation group; and a third transition matrix determination unit, used to use the transition matrix between the first and second target aggregation groups as the transition matrix between the first target road element in the first road image and the second target road element in the second road image; wherein the second target road element refers to the road element in the second road image whose feature point is located in the second target aggregation group; and the first road image and the second road image are different road images acquired from two acquisition points with an interval distance not exceeding a distance threshold. The semantic association module is used to semantically associate road elements in the first road image with road elements in the second road image based on the transition matrix corresponding to the road elements that match the feature points. The two road elements that are semantically associated represent the same road element in space. The three-dimensional spatial information determination module is used to determine the three-dimensional spatial information corresponding to the semantically associated road element based on the position information of multiple feature points in the first road image and the position information in the second road image. The map building module is used to build a three-dimensional map based on the three-dimensional spatial information of the semantically associated road elements and the element type to which the semantically associated road elements belong.

13. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, the method as described in any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 11.

Images