Map data expansion method, device, electronic device, medium and program product

By determining the matching point pairs and poses in the image and map data, and converting two-dimensional coordinate points into three-dimensional coordinate points, the problem of insufficient expansion accuracy in traditional technology is solved, and more accurate expansion of map data and user experience is achieved promote.

CN113535875BActive Publication Date: 2024-11-01CHONGQING LIQI TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202110797220.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-14
Publication Date
2024-11-01
Estimated Expiration
2041-07-14

AI Technical Summary

Technical Problem

Traditional map data expansion technology is insufficient in accuracy and cannot provide accurate map data, resulting in reduced or invalid map reliability in environments such as parking lots.

Method used

Convert 2D coordinate points to available by obtaining the 2D coordinate points and feature description subset associated with the image set, as well as the 3D coordinate points and feature description subset in the map data, determine the matching point pair and calculate the pose, and convert the 2D coordinate points into available Three-dimensional coordinate points of the expanded map data.

Benefits of technology

It realizes high-precision expansion of map data, provides more accurate map data, and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113535875B_ABST
    Figure CN113535875B_ABST
Patent Text Reader

Abstract

The present disclosure provides a map data expansion method, device, electronic device, medium and program product, which relate to the computer field, especially to the map drawing field and the autonomous driving field, and can be applied to assisted driving and autonomous driving scenarios. The method includes: obtaining a first set of two-dimensional coordinate points and a first feature descriptor set associated with a first image set; obtaining map data, including a set of three-dimensional coordinate points and a second feature descriptor set; determining a first matching point pair set between a first subset of two-dimensional coordinate points and a set of three-dimensional coordinate points associated with a first image in the first image set; determining the first pose of the first image relative to the set of three-dimensional coordinate points based on the first matching point pair set; and converting the first subset of two-dimensional coordinate points into the first subset of three-dimensional coordinate points based on the first pose. Using the above method, the expansion of map data can be achieved with higher accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computers, in particular to the fields of map drawing and autonomous driving, and can be applied to assisted driving and autonomous driving scenarios, and more specifically, to a map data expansion method, a map data expansion device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] For assisted driving or autonomous driving applications that provide autonomous parking functions, for example, it is necessary to calculate the vehicle's position in real time based on a pre-built parking map. However, the parking environment may change slowly over time, especially in outdoor parking lots. Different weather or different lighting conditions at different times of the day can also cause the map to become less reliable or fail, which can reduce the reliability of the built map. Therefore, sensor data can be collected when the user passes through the mapping area, and the collected sensor data can be used to expand and update the map data, so that the map can automatically adapt to changes in the scene, thereby extending the available time of the route map.

[0003] However, conventional techniques for map data augmentation have insufficient data augmentation accuracy and thus cannot provide accurate map data. Summary of the invention

[0004] According to an embodiment of the present disclosure, a map data expansion method, a map data expansion device, an electronic device, a computer-readable storage medium, and a computer program product are provided.

[0005] In a first aspect of the present disclosure, a map data expansion method is provided, comprising: obtaining a first set of two-dimensional coordinate points associated with a first image set and a first feature descriptor set associated with the first set of two-dimensional coordinate points; obtaining map data, the map data comprising a set of three-dimensional coordinate points and a second feature descriptor set associated with the set of three-dimensional coordinate points, the first image set and the map data being for the same map; for a first image in the first image set, determining a first set of matching point pairs between a first subset of two-dimensional coordinate points associated with the first image and the set of three-dimensional coordinate points based on the first feature descriptor set and the second feature descriptor set; determining a first pose of the first image relative to the set of three-dimensional coordinate points based on the first matching point pair set; and based on the first pose, converting the first subset of two-dimensional coordinate points into a first subset of three-dimensional coordinate points that can be used to expand the map data.

[0006] In a second aspect of the present disclosure, a map data expansion device is provided, comprising: a first acquisition module, configured to perform feature extraction on a first image set to obtain a first set of two-dimensional coordinate points and a first feature descriptor set associated with the first set of two-dimensional coordinate points; a second acquisition module, configured to acquire map data, the map data comprising a set of three-dimensional coordinate points and a second feature descriptor set associated with the set of three-dimensional coordinate points, the first image set and the map data being for the same map; a first matching point pair set determination module, configured to determine, for a first image in the first image set, a first matching point pair set between a first subset of two-dimensional coordinate points associated with the first image and a set of three-dimensional coordinate points based on the first feature descriptor set and the second feature descriptor set; a first pose determination module, configured to determine a first pose of the first image relative to the set of three-dimensional coordinate points based on the first matching point pair set; and a conversion module, configured to convert the first subset of two-dimensional coordinate points into a first subset of three-dimensional coordinate points that can be used to expand the map data based on the first pose.

[0007] In the third aspect of the present disclosure, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to implement the method according to the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, and when the computer program is executed by a processor, the computer program performs the method according to the first aspect of the present disclosure.

[0010] Using the technology according to the present application, a method for expanding map data is provided. Using the technical solution of this method, map data can be expanded with higher precision, thereby not only providing more accurate map data, but also improving the user experience of users who need to use the expanded map data.

[0011] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the exemplary embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present disclosure. It should be understood that the accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:

[0013] Figure 1 A schematic block diagram of a map data augmentation environment 100 is shown in which the map data augmentation method in some embodiments of the present disclosure may be implemented;

[0014] Figure 2 A flowchart of a map data expansion method 200 according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A flowchart of a map data expansion method 300 according to an embodiment of the present disclosure is shown;

[0016] Figure 4 A schematic block diagram of a map data expansion device 400 according to an embodiment of the present disclosure is shown; and

[0017] Figure 5 A schematic block diagram of an example electronic device 500 that may be used to implement embodiments of the present disclosure is shown.

[0018] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0020] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] As described above in the background art, conventional technologies for map data expansion have insufficient data expansion accuracy and thus cannot provide accurate map data.

[0022] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, an embodiment of the present disclosure proposes a map data expansion method. Using the technical solution of this method, map data can be expanded with higher precision, thereby not only providing more accurate map data, but also improving the user experience of users who need to use the expanded map data.

[0023] Figure 1 FIG. 1 is a schematic block diagram of a map data augmentation environment 100 in which the map data augmentation method in some embodiments of the present disclosure may be implemented. According to one or more embodiments of the present disclosure, the map data augmentation environment 100 may be a cloud environment. Figure 1 As shown in , the map data augmentation environment 100 includes a computing device 110. In the map data augmentation environment 100, augmentation-related data 120 is provided to the computing device 110 as an input of the computing device 110. The augmentation-related data 120 may include, for example, a first set of images, a first set of two-dimensional coordinate points associated with the first set of images, a first set of feature descriptors associated with the first set of two-dimensional coordinate points, and map data, wherein the map data includes a set of three-dimensional coordinate points and a second set of feature descriptors associated with the set of three-dimensional coordinate points.

[0024] According to one or more embodiments of the present disclosure, a set of three-dimensional coordinate points, which may also be referred to as a point cloud, may be a set of points with three-dimensional coordinates, where the three-dimensional coordinates of the points may be, for example, world coordinates in the form of (x, y, z).

[0025] According to one or more embodiments of the present disclosure, a feature descriptor set such as a first feature descriptor set and a second feature descriptor set includes feature descriptors, wherein the feature descriptors may be in the form of a multidimensional vector. Feature descriptors are derived from feature points in an image set collected by an image acquisition device, and feature points are local features of an image, which remain invariant to rotation, resizing, and brightness changes, and also maintain a certain degree of stability to perspective changes, affine transformations, and noise. A feature descriptor set may be obtained by extracting features from images in the collected image set using a feature extraction algorithm such as SIFT, SUPERPOINT, or ORB. Since feature points in an image have corresponding feature descriptors, the three-dimensional coordinate points in a set of three-dimensional coordinate points included in map data for constructing a map using an image set will inherit the feature descriptors of the corresponding feature points in the image set.

[0026] According to one or more embodiments of the present disclosure, the first image set and the map data are for the same map. For example, the first image set may be an image set collected for a parking lot scene, and the map data may be map data of a three-dimensional map constructed for the same parking lot scene.

[0027] According to one or more embodiments of the present disclosure, the first image set may be a set of images recently acquired through a collection process, and the map data may be map data of a currently existing map, wherein the map data may be larger map data formed through multiple map data expansions.

[0028] After obtaining the expanded relevant data 120 as input, the computing device 110 can expand the map data based on the feature descriptor set included in the expanded relevant data 120 by determining a set of matching point pairs, determining a pose, and other operations using the first image set included in the expanded relevant data 120, thereby obtaining expanded map data and outputting the expanded map data.

[0029] It should be understood that the map data augmentation environment 100 is merely illustrative and not restrictive, and is expandable, and may include more computing devices 110, and may provide more augmented related data 120 as input to the computing devices 110, thereby satisfying the needs of more users to utilize more computing devices 110 at the same time, and even utilize more augmented related data 120 to perform map data augmentation operations simultaneously or non-simultaneously.

[0030] exist Figure 1 In the illustrated map data augmentation environment 100 , the operation of inputting augmentation-related data 120 into the computing device 110 may be performed through a network, and the computing device 110 may be, for example, a computing device in a high-computing-power cloud platform.

[0031] Figure 2 FIG. 2 shows a flow chart of a map data expansion method 200 according to an embodiment of the present disclosure. Specifically, the map data expansion method 200 may be composed of Figure 1 The map data augmentation method 200 is executed by the computing device 110 in the map data augmentation environment 100 shown in FIG. It should be understood that the map data augmentation method 200 may further include additional operations not shown and / or may omit the operations shown, and the scope of the present disclosure is not limited in this respect.

[0032] In block 202, computing device 110 obtains a first set of two-dimensional coordinate points associated with a first set of images and a first set of feature descriptors associated with the first set of two-dimensional coordinate points. The first set of images may be collected by an image collection device for a scene for which map data needs to be constructed.

[0033] According to one or more embodiments of the present disclosure, the computing device 110 may first acquire a first set of images. Then, the computing device 110 may perform feature extraction on the first set of acquired images using a feature extraction algorithm such as SIFT, SUPERPOINT, or ORB to acquire a first set of two-dimensional coordinate points, which may be a set of feature points, and a first set of feature descriptors associated with the first set of two-dimensional coordinate points.

[0034] In block 204, the computing device 110 obtains map data. According to one or more embodiments of the present disclosure, the map data includes a set of three-dimensional coordinate points and a second feature descriptor set associated with the set of three-dimensional coordinate points, and the first image set and the map data are for the same map. In some embodiments, the map data may be map data constructed based on a set of images obtained by a single acquisition of a scene. In other embodiments, the map data may be larger map data that has been formed after multiple map data expansions.

[0035] At block 206 , computing device 110 determines, for a first image in the first set of images, a first set of matching point pairs between a first subset of two-dimensional coordinate points and a set of three-dimensional coordinate points associated with the first image based on the first set of feature descriptors and the second set of feature descriptors.

[0036] According to one or more embodiments of the present disclosure, the computing device 110 may first determine, for a first two-dimensional coordinate point in a first subset of two-dimensional coordinate points associated with a first image, a second feature descriptor in a second feature descriptor set that is closest to the first feature descriptor based on a first feature descriptor in a first feature descriptor set that is associated with the first two-dimensional coordinate point and a second feature descriptor set. As described above, a feature descriptor may be in the form of a multidimensional vector, so the similarity between the two feature descriptors may be determined by the distance between the two feature descriptors, wherein the closer the distance between the two feature descriptors, the higher the similarity between the two feature descriptors.

[0037] Then, the computing device 110 may determine whether the distance between the first feature descriptor and the second feature descriptor is less than a first threshold distance, and when it is determined that the distance between the first feature descriptor and the second feature descriptor is less than the first threshold distance, determine the point pair associated with the first feature descriptor and the second feature descriptor as a matching point pair in the first matching point pair set.

[0038] According to one or more embodiments of the present disclosure, the first image in the first image set may refer to any image in the first image set. When determining a first set of matching point pairs between a first subset of two-dimensional coordinate points and a set of three-dimensional coordinate points associated with the first image, the computing device 110 may search for a matching three-dimensional coordinate point in the set of three-dimensional coordinate points for each point in the first subset of two-dimensional coordinate points to form a matching point pair, and the matching point pairs formed for all points in the first subset of two-dimensional coordinate points constitute the first set of matching point pairs. In some embodiments, only a few points in the first subset of two-dimensional coordinate points can form matching point pairs with three-dimensional coordinate points in the set of three-dimensional coordinate points.

[0039] According to one or more embodiments of the present disclosure, the two-dimensional coordinate point included in the matching point pair can be preliminarily considered to be the same three-dimensional coordinate point in the physical world as the three-dimensional coordinate point included in the matching point pair after being converted into a three-dimensional coordinate point in map data.

[0040] In block 208, the computing device 110 determines a first pose of the first image relative to the set of three-dimensional coordinate points based on the first set of matching point pairs determined in block 206. According to one or more embodiments of the present disclosure, the computing device 110 determines a first pose of the first image relative to the set of three-dimensional coordinate points based on the first set of matching point pairs using a perspective-n-point (PNP) algorithm.

[0041] According to one or more embodiments of the present disclosure, the computing device 110 may determine whether the number of matching point pairs in a first matching point pair set associated with a first image is greater than the number of matching point pairs in each matching point pair set associated with each other image in the first image set whose pose has not yet been determined, and upon determining that the number of matching point pairs in the first matching point pair set associated with the first image is greater than the number of matching point pairs in each matching point pair set associated with each other image in the first image set whose pose has not yet been determined, determine a first pose based on the first matching point pair set.

[0042] According to one or more embodiments of the present disclosure, when determining the first pose, the computing device 110 may first determine the second pose of the first image relative to the set of three-dimensional coordinate points based on the first set of matching point pairs. Then, the computing device 110 may optimize the second pose using a bundle adjustment nonlinear optimization algorithm based on a reprojection error constraint to obtain the first pose.

[0043] According to one or more embodiments of the present disclosure, when optimizing the second posture, the computing device 110 may be based on at least one of the following in addition to the reprojection error constraint: a posture constraint associated with two adjacent images in the first image set; a rotation angle constraint associated with two adjacent images; and a point-to-point distance error constraint associated with two adjacent images. The posture constraint associated with two adjacent images in the first image set may be associated with the wheel speed or the interval distance when the two adjacent images are captured, wherein the interval distance may be obtained by integrating the wheel speed and time. The rotation angle constraint associated with two adjacent images may be associated with the motion acceleration, angular velocity, etc. of the object, and may be measured by an inertial sensor, thereby obtaining a posture angle used to represent the rotation angle between the two images. The point-to-point distance error constraint associated with two adjacent images may be associated with the same point in the two images, such as the same parking space corner point.

[0044] In other words, the steps involved in box 208 can be performed iteratively, and each time the image with the largest number of matching point pairs in the associated matching point pair set among the multiple images in the first image set whose poses have not yet been determined.

[0045] At block 210, the computing device 110 converts the first subset of two-dimensional coordinate points into a first subset of three-dimensional coordinate points that can be used to augment the map data based on the first pose determined at block 208. According to one or more embodiments of the present disclosure, the computing device 110 may convert each two-dimensional coordinate point in the first subset of two-dimensional coordinate points associated with the first image into a corresponding three-dimensional coordinate point in the set of three-dimensional coordinate points associated with the map data based on the first pose of the first image relative to the set of three-dimensional coordinate points. The converted corresponding three-dimensional coordinate point can then be added to the set of three-dimensional coordinate points associated with the map data to augment the map data.

[0046] According to one or more embodiments of the present disclosure, method 200 may be performed iteratively until a three-dimensional coordinate point that can be used to expand map data is determined based on each image in the first image set.

[0047] It can be seen that the map data expansion method 200 mainly involves the matching between the two-dimensional coordinate points associated with the first image set and the three-dimensional coordinate points associated with the map data.

[0048] Figure 3 FIG. 3 is a flowchart of a map data expansion method 300 according to an embodiment of the present disclosure. Specifically, the map data expansion method 300 may also be Figure 1The map data augmentation method 300 is executed by the computing device 110 in the map data augmentation environment 100 shown in FIG. It should be understood that the map data augmentation method 300 may also include additional operations not shown and / or may omit the operations shown, and the scope of the present disclosure is not limited in this respect. The map data augmentation method 300 may be considered as an extension of the map data augmentation method 200.

[0049] In block 302, computing device 110 obtains a first set of two-dimensional coordinate points associated with a first image set and a first set of feature descriptors associated with the first set of two-dimensional coordinate points. The specific content of block 302 is the same as that of block 202 and is not described again.

[0050] In block 304, the computing device 110 obtains map data. According to one or more embodiments of the present disclosure, the map data includes a set of three-dimensional coordinate points and a second feature descriptor set associated with the set of three-dimensional coordinate points, and the first image set and the map data are for the same map. The specific content of block 304 is the same as that of block 204, and will not be repeated here.

[0051] In block 306, the computing device 110 determines, for the first image in the first image set, a first set of matching point pairs between a first subset of two-dimensional coordinate points and a set of three-dimensional coordinate points associated with the first image based on the first feature descriptor set and the second feature descriptor set. The specific content of block 306 is the same as that of block 206, and is not described in detail herein.

[0052] In block 308, computing device 110 determines a first pose of the first image relative to the set of three-dimensional coordinate points based on the first set of matching point pairs determined in block 306. The specific content of block 308 is the same as that of block 208 and will not be described in detail herein.

[0053] In block 310, computing device 110 converts the first subset of two-dimensional coordinate points into a first subset of three-dimensional coordinate points that can be used to augment the map data based on the first pose determined in block 308. The specific content of block 310 is the same as that of block 210 and is not described again.

[0054] In box 312, the computing device 110 determines a plurality of other images in the first image set that are spaced from the first image by a plurality of predetermined intervals. According to one or more embodiments of the present disclosure, the plurality of predetermined intervals may include 1 image, 3 images, 7 images, 9 images, 12 images, etc. When determining the aforementioned plurality of other images, if starting from the first image in the first image set, only the images spaced by the aforementioned predetermined intervals after the first image need to be considered. At the same time, when determining, for example, the third image, since the first image and the third image have been matched when determining the first image, in fact, only the images spaced by the aforementioned predetermined intervals after the third image need to be considered.

[0055] At block 314 , the computing device 110 determines, based on the first set of feature descriptors, a second set of matching point pairs between the first subset of two-dimensional coordinate points associated with the first image and a plurality of subsets of two-dimensional coordinate points associated with a plurality of other images.

[0056] According to one or more embodiments of the present disclosure, the computing device 110 may first determine, for a first two-dimensional coordinate point in a first subset of two-dimensional coordinate points, based on a first feature descriptor set, a third feature descriptor in a first subset of feature descriptors associated with a second image in a plurality of other images, which is closest to a first feature descriptor associated with the first two-dimensional coordinate point.

[0057] Then, the computing device 110 may determine whether the distance between the first feature descriptor and the third feature descriptor is less than a second threshold distance, and when it is determined that the distance between the first feature descriptor and the third feature descriptor is less than the second threshold distance, determine the point pair associated with the first feature descriptor and the third feature descriptor as a matching point pair in the second matching point pair set.

[0058] According to one or more embodiments of the present disclosure, the first two-dimensional coordinate point in the first subset of two-dimensional coordinate points may be any two-dimensional coordinate point in the first subset of two-dimensional coordinate points. When determining a second set of matching point pairs between the first subset of two-dimensional coordinate points associated with the first image and multiple subsets of two-dimensional coordinate points associated with multiple other images, the computing device 110 may search for matching two-dimensional coordinate points in multiple subsets of two-dimensional coordinate points associated with multiple other images for each point in the first subset of two-dimensional coordinate points to form a matching point pair, and the matching point pairs formed for all points in the first subset of two-dimensional coordinate points constitute the second set of matching point pairs. In some embodiments, since multiple subsets of two-dimensional coordinate points are associated with multiple other images, the first two-dimensional coordinate point in the first subset of two-dimensional coordinate points may form more than one matching point pair with two-dimensional coordinate points associated with more than one other image in the multiple other images. In some embodiments, only a few points in the first subset of two-dimensional coordinate points can form matching point pairs with three-dimensional coordinate points in the set of three-dimensional coordinate points.

[0059] At block 316, the computing device 110 determines whether an image associated with any point in the first pair of matching points in the second set of matching point pairs has been posed. If an image associated with any point in the first pair of matching points in the second set of matching point pairs has been posed, the method 300 proceeds to block 318. Otherwise, the method 300 may not perform an operation.

[0060] According to one or more embodiments of the present disclosure, if the pose of an image associated with any point in a first matching point pair in a second matching point pair set has been determined, it means that at least one point in this matching point pair has been matched with a set of three-dimensional coordinate points associated with map data. At this time, this matching point pair can be used to expand the map data.

[0061] At block 318, computing device 110 generates a second subset of three-dimensional coordinate points that can be used to augment the map data based on the first matching point pair by triangulating the feature points. According to one or more embodiments of the present disclosure, triangulating the feature points on the first matching point pair can be considered as visual reconstruction of the map data from the first matching points.

[0062] In block 320, the computing device 110 determines a subset of three-dimensional coordinate points in the first set of matching point pairs that are associated with the set of three-dimensional coordinate points. According to one or more embodiments of the present disclosure, since the first set of matching point pairs is a set of matching point pairs between the first subset of two-dimensional coordinate points associated with the first image and the set of three-dimensional coordinate points, each matching point pair in the first set of matching point pairs includes a point in the set of three-dimensional coordinate points associated with the map data. Therefore, a subset of three-dimensional coordinate points in the first set of matching point pairs that are associated with the set of three-dimensional coordinate points can be determined.

[0063] In block 322, the computing device 110 determines a second set of images associated with the subset of three-dimensional coordinate points for forming map data. According to one or more embodiments of the present disclosure, map data is also formed by a set of images, so each point in the set of three-dimensional coordinate points associated with the map data is associated with at least one image for forming the map data. Therefore, the second set of images for forming the map data may be determined based on the subset of three-dimensional coordinate points determined in block 320, and the second set of images may not be a set of all images for forming the map data, but may only include a portion of all images for forming the map data.

[0064] At block 324 , the computing device 110 determines, based on the first set of feature descriptors, a third set of matching point pairs between the first subset of two-dimensional coordinate points associated with the first image and the subset of two-dimensional coordinate points associated with the second set of images.

[0065] According to one or more embodiments of the present disclosure, the computing device 110 may first determine, for a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, based on the first feature descriptor set, a fourth feature descriptor in the second subset of feature descriptors associated with a third image in the second image set, which is closest to the first feature descriptor associated with the first two-dimensional coordinate point.

[0066] Then, the computing device 110 may determine whether the distance between the first feature descriptor and the fourth feature descriptor is less than a third threshold distance, and when it is determined that the distance between the first feature descriptor and the fourth feature descriptor is less than the third threshold distance, determine the point pair associated with the first feature descriptor and the fourth feature descriptor as a matching point pair in the third matching point pair set.

[0067] According to one or more embodiments of the present disclosure, the first two-dimensional coordinate point in the first subset of two-dimensional coordinate points may be any two-dimensional coordinate point in the first subset of two-dimensional coordinate points. When determining a third set of matching point pairs between the first subset of two-dimensional coordinate points associated with the first image and the subset of two-dimensional coordinate points associated with the second image set, the computing device 110 may search for a matching two-dimensional coordinate point in the subset of two-dimensional coordinate points associated with the second image set for each point in the first subset of two-dimensional coordinate points to form a matching point pair, and the matching point pairs formed for all points in the first subset of two-dimensional coordinate points constitute the third set of matching point pairs. In some embodiments, since the subset of two-dimensional coordinate points is associated with multiple images, the first two-dimensional coordinate point in the first subset of two-dimensional coordinate points may form more than one matching point pair with two-dimensional coordinate points associated with more than one image in the second image set. In some embodiments, only a few points in the first subset of two-dimensional coordinate points can form matching point pairs with two-dimensional coordinate points in the subset of two-dimensional coordinate points.

[0068] In block 326, the computing device 110 generates a third subset of three-dimensional coordinate points that can be used to expand the map data based on the third set of matching point pairs by triangulating the feature points. According to one or more embodiments of the present disclosure, triangulating the feature points of the matching point pairs in the third set of matching point pairs can be considered as visual reconstruction of the map data from the third set of matching point pairs.

[0069] At block 328, computing device 110 uses the bundle adjustment nonlinear optimization algorithm to optimize at least one of the following: the first subset of three-dimensional coordinate points, the second subset of three-dimensional coordinate points, and the third subset of three-dimensional coordinate points. The manner of using the bundle adjustment nonlinear optimization algorithm for optimization is the same as that described in block 208 of reference method 200, and will not be repeated here.

[0070] In block 330 , the computing device 110 reprojects a first three-dimensional coordinate point in the first subset of three-dimensional coordinate points to obtain a reprojected two-dimensional coordinate point.

[0071] At block 332, the computing device 110 determines whether a distance between the reprojected two-dimensional coordinate point and a second two-dimensional coordinate point in the first subset of two-dimensional coordinate points that is associated with the first three-dimensional coordinate point is greater than a fourth threshold distance. If the distance between the reprojected two-dimensional coordinate point and the second two-dimensional coordinate point in the first subset of two-dimensional coordinate points that is associated with the first three-dimensional coordinate point is greater than the fourth threshold distance, the method 300 proceeds to block 334. Otherwise, the method 300 may not perform an operation.

[0072] According to one or more embodiments of the present disclosure, when the distance between a reprojected two-dimensional coordinate point and a second two-dimensional coordinate point in the first subset of two-dimensional coordinate points and associated with the first three-dimensional coordinate point is greater than a fourth threshold distance, it indicates that the first three-dimensional coordinate point used to generate the reprojected two-dimensional coordinate point may be erroneous and therefore needs to be further processed.

[0073] At block 334 , computing device 110 deletes the first three-dimensional coordinate point from the first subset of three-dimensional coordinate points.

[0074] According to one or more embodiments of the present disclosure, when the distance between a reprojected two-dimensional coordinate point and a second two-dimensional coordinate point in the first subset of two-dimensional coordinate points and associated with the first three-dimensional coordinate point is greater than a fourth threshold distance, the first three-dimensional coordinate point may not be directly deleted from the first subset of three-dimensional coordinate points. Instead, it may be decided whether the first three-dimensional coordinate point needs to be deleted, for example, by manual judgment.

[0075] According to one or more embodiments of the present disclosure, the steps in blocks 330 to 334 may also be applicable to the second subset of three-dimensional coordinate points and the third subset of three-dimensional coordinate points.

[0076] According to one or more embodiments of the present disclosure, method 300 may be performed iteratively until a three-dimensional coordinate point that can be used to expand map data is determined based on each image in the first image set.

[0077] References Figures 1 to 3 The following describes the map data augmentation environment 100 in which the map data augmentation method in some embodiments of the present disclosure can be implemented, the map data augmentation method 200 according to an embodiment of the present disclosure, and the map data augmentation method 300 according to an embodiment of the present disclosure. It should be understood that the above description is for better illustrating the contents recorded in the present disclosure, and is not intended to be limiting in any way.

[0078] It should be understood that the numbers and sizes of various elements used in the above-mentioned drawings of the present disclosure are only examples, and are not intended to limit the scope of protection of the present disclosure. The above-mentioned numbers and sizes can be arbitrarily set as needed without affecting the normal implementation of the embodiments of the present disclosure.

[0079] See above Figures 1 to 3 The details of the map data expansion method 200 and the map data expansion method 300 according to the embodiment of the present disclosure are described. Figure 4 Describe the various modules in the map data expansion device.

[0080] Figure 4 FIG. 4 is a schematic block diagram of a map data expansion device 400 according to an embodiment of the present disclosure. Figure 4 As shown, the map data expansion device 400 may include: a first acquisition module 410, configured to perform feature extraction on the first image set to obtain a first set of two-dimensional coordinate points and a first feature description subset associated with the first set of two-dimensional coordinate points; a second acquisition module 420, configured to acquire map data, the map data including a set of three-dimensional coordinate points and a second feature description subset associated with the set of three-dimensional coordinate points, the first image set and the map data being for the same map; a first matching point pair set determination module 430, configured to determine, for a first image in the first image set, a first matching point pair set between a first subset of two-dimensional coordinate points associated with the first image and a set of three-dimensional coordinate points based on the first feature description subset and the second feature description subset; a first pose determination module 440, configured to determine a first pose of the first image relative to the set of three-dimensional coordinate points based on the first matching point pair set; and a conversion module 450, configured to convert the first subset of two-dimensional coordinate points into a first subset of three-dimensional coordinate points that can be used to expand the map data based on the first pose.

[0081] In one or more embodiments, the first matching point pair set determination module 430 includes: a first feature descriptor determination module (not shown), configured to determine, for a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, a second feature descriptor in the second feature descriptor set that is closest to the first feature descriptor based on a first feature descriptor in the first feature descriptor set that is associated with the first two-dimensional coordinate point, and a second feature descriptor set; and a first matching point pair determination module (not shown), configured to determine a point pair associated with the first feature descriptor and the second feature descriptor as a matching point pair in the first matching point pair set if the distance between the first feature descriptor and the second feature descriptor is less than a first threshold distance.

[0082] In one or more embodiments, the first pose determination module 440 includes: a second pose determination module (not shown), configured to determine the first pose based on the first matching point pair set if the number of matching point pairs in the first matching point pair set associated with the first image is greater than the number of matching point pairs in each matching point pair set associated with each other image in the first image set whose pose has not yet been determined.

[0083] In one or more embodiments, the map data expansion device 400 also includes: a first image determination module (not shown), configured to determine a plurality of other images in the first image set that are spaced apart from the first image by a plurality of predetermined intervals; a second matching point pair determination module (not shown), configured to determine, based on the first feature descriptor set, a second matching point pair set between a first subset of two-dimensional coordinate points associated with the first image and a plurality of subsets of two-dimensional coordinate points associated with the plurality of other images; and a first subset generation module (not shown), configured to generate a second subset of three-dimensional coordinate points that can be used to expand the map data based on the first matching point pair by triangulating the feature points if the image associated with any point in the first matching point pair in the second matching point pair set has its pose determined.

[0084] In one or more embodiments, the second matching point pair determination module includes: a second feature descriptor determination module (not shown), configured to determine, for a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, based on the first feature descriptor set, a third feature descriptor in the first subset of feature descriptors associated with a second image in multiple other images, which is closest to the first feature descriptor associated with the first two-dimensional coordinate point; and a third matching point pair determination module (not shown), configured to determine the point pair associated with the first feature descriptor and the third feature descriptor as a matching point pair in the second matching point pair set if the distance between the first feature descriptor and the third feature descriptor is less than a second threshold distance.

[0085] In one or more embodiments, the map data expansion device 400 also includes: a subset determination module (not shown), configured to determine a subset of three-dimensional coordinate points in the first matching point pair set that is associated with the set of three-dimensional coordinate points; an image set determination module (not shown), configured to determine a second image set associated with the subset of three-dimensional coordinate points for forming map data; a fourth matching point pair determination module (not shown), configured to determine a third matching point pair set between the first subset of two-dimensional coordinate points associated with the first image and the subset of two-dimensional coordinate points associated with the second image set based on the first feature description subset; and a second subset generation module (not shown), configured to generate a third subset of three-dimensional coordinate points that can be used to expand map data based on the third matching point pair set by triangulating feature points.

[0086] In one or more embodiments, the fourth matching point pair determination module includes: a third feature descriptor determination module (not shown), configured to determine, for a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, based on the first feature descriptor set, a fourth feature descriptor in the second subset of feature descriptors associated with a third image in the second image set, which is closest to the first feature descriptor associated with the first two-dimensional coordinate point; and a fifth matching point pair determination module (not shown), configured to determine the point pair associated with the first feature descriptor and the fourth feature descriptor as a matching point pair in the third matching point pair set if the distance between the first feature descriptor and the fourth feature descriptor is less than a third threshold distance.

[0087] In one or more embodiments, the first pose determination module 440 includes: a third pose determination module (not shown), configured to determine a second pose of the first image relative to a set of three-dimensional coordinate points based on a first set of matching point pairs; and a first optimization module (not shown), configured to optimize the second pose using a bundle adjustment nonlinear optimization algorithm based on a reprojection error constraint to obtain the first pose.

[0088] In one or more embodiments, the first optimization module is further configured to optimize the second pose using a bundle adjustment nonlinear optimization algorithm based on at least one of: a pose constraint associated with two adjacent images in the first image set; a rotation angle constraint associated with the two adjacent images; and a point pair distance error constraint associated with the two adjacent images.

[0089] In one or more embodiments, the map data expansion device 400 further includes: a second optimization module (not shown) configured to optimize the first subset of three-dimensional coordinate points using a bundle adjustment nonlinear optimization algorithm based on a reprojection error constraint.

[0090] In one or more embodiments, the map data expansion device 400 also includes: a reprojection module (not shown), configured to reproject a first three-dimensional coordinate point in the first subset of three-dimensional coordinate points to obtain a reprojected two-dimensional coordinate point; and a deletion module (not shown), configured to delete the first three-dimensional coordinate point from the first subset of three-dimensional coordinate points if the distance between the reprojected two-dimensional coordinate point and a second two-dimensional coordinate point in the first subset of two-dimensional coordinate points that is associated with the first three-dimensional coordinate point is greater than a fourth threshold distance.

[0091] Through the above reference Figures 1 to 4According to the description, the technical solution of the embodiment of the present disclosure has many advantages over the traditional solution. For example, by using the technical solution of the method, the map data can be expanded with higher precision, thereby not only providing more accurate map data, but also improving the user experience of users who need to use the expanded map data.

[0092] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

[0093] According to an embodiment of the present disclosure, an electronic device, a computer-readable storage medium, and a computer program product are also provided.

[0094] Figure 5 Schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. Figure 1 The computing device 110 shown and Figure 4 The map data expansion device 400 shown can be implemented by an electronic device 500. The electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0095] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0096] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0097] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the map data expansion methods 200 and 300. For example, in some embodiments, the map data expansion methods 200 and 300 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the map data expansion methods 200 and 300 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the map data expansion methods 200 and 300 in any other appropriate manner (eg, by means of firmware).

[0098] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0102] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0103] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A map data expansion method, comprising: Acquire a first set of two-dimensional coordinate points associated with a first image set and a first set of feature descriptors associated with the first set of two-dimensional coordinate points; Acquire map data, the map data comprising a set of three-dimensional coordinate points and a second feature descriptor set associated with the set of three-dimensional coordinate points, wherein the first image set and the map data are for the same map; For a first image in the first image set, determining, based on the first feature descriptor set and the second feature descriptor set, a first set of matching point pairs between a first subset of two-dimensional coordinate points associated with the first image and the set of three-dimensional coordinate points; Determine, based on the first set of matching point pairs, a first pose of the first image relative to the set of three-dimensional coordinate points; Based on the first pose, converting the first subset of two-dimensional coordinate points into a first subset of three-dimensional coordinate points that can be used to expand the map data; determining a plurality of other images in the first image set that are spaced apart from the first image by a plurality of predetermined intervals; Based on the first feature descriptor set, determining a second set of matching point pairs between the first subset of two-dimensional coordinate points associated with the first image and multiple subsets of two-dimensional coordinate points associated with the multiple other images; as well as If the image associated with any point in the first matching point pair in the second matching point pair set has been posed, a second subset of three-dimensional coordinate points that can be used to expand the map data is generated based on the first matching point pair by triangulating feature points.

2. The method according to claim 1, wherein determining the first set of matching point pairs comprises: For a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, based on a first feature descriptor in the first feature descriptor set that is associated with the first two-dimensional coordinate point and the second feature descriptor set, determining a second feature descriptor in the second feature descriptor set that is closest to the first feature descriptor; and If the distance between the first feature descriptor and the second feature descriptor is less than a first threshold distance, the point pair associated with the first feature descriptor and the second feature descriptor is determined as a matching point pair in the first matching point pair set.

3. The method of claim 1 , wherein determining the first pose comprises: If the number of matching point pairs in the first set of matching point pairs associated with the first image is greater than the number of matching point pairs in each set of matching point pairs associated with each other image in the first set of images whose pose has not yet been determined, the first pose is determined based on the first set of matching point pairs.

4. The method according to claim 1, wherein determining the second set of matching point pairs comprises: For a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, determining, based on the first set of feature descriptors, a third feature descriptor in the first subset of feature descriptors associated with a second image in the plurality of other images, which is closest to the first feature descriptor associated with the first two-dimensional coordinate point; and If the distance between the first feature descriptor and the third feature descriptor is less than a second threshold distance, the point pair associated with the first feature descriptor and the third feature descriptor is determined as a matching point pair in the second matching point pair set.

5. The method according to claim 1, further comprising: Determine a subset of three-dimensional coordinate points in the first set of matching point pairs that are associated with the set of three-dimensional coordinate points; determining a second set of images associated with the subset of the three-dimensional coordinate points for forming the map data; determining, based on the first set of feature descriptors, a third set of matching point pairs between the first subset of two-dimensional coordinate points associated with the first image and the subset of two-dimensional coordinate points associated with the second set of images; as well as A third subset of three-dimensional coordinate points that can be used to expand the map data is generated based on the third set of matching point pairs through feature point triangulation.

6. The method according to claim 5, wherein determining the third set of matching point pairs comprises: For a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, determining, based on the first set of feature descriptors, a fourth feature descriptor in the second subset of feature descriptors associated with a third image in the second set of images, which is closest to the first feature descriptor associated with the first two-dimensional coordinate point; and If the distance between the first feature descriptor and the fourth feature descriptor is less than a third threshold distance, the point pair associated with the first feature descriptor and the fourth feature descriptor is determined as a matching point pair in the third matching point pair set.

7. The method of claim 1 , wherein determining the first pose comprises: Determine a second pose of the first image relative to the set of three-dimensional coordinate points based on the first set of matching point pairs; as well as Based on the reprojection error constraint, a bundle adjustment nonlinear optimization algorithm is used to optimize the second pose to obtain the first pose.

8. The method of claim 7, wherein optimizing the second pose is further based on at least one of: a pose constraint associated with two adjacent images in the first image set; a rotation angle constraint associated with the two adjacent images; and A point pair distance error constraint associated with the two adjacent images.

9. The method according to claim 7, further comprising: Based on the reprojection error constraint, the first subset of three-dimensional coordinate points is optimized using the bundle adjustment nonlinear optimization algorithm.

10. The method according to claim 1, further comprising: Reprojecting a first three-dimensional coordinate point in the first subset of three-dimensional coordinate points to obtain a reprojected two-dimensional coordinate point; as well as If the distance between the reprojected two-dimensional coordinate point and a second two-dimensional coordinate point in the first subset of two-dimensional coordinate points associated with the first three-dimensional coordinate point is greater than a fourth threshold distance, the first three-dimensional coordinate point is deleted from the first subset of three-dimensional coordinate points.

11. A map data expansion device, comprising: A first acquisition module is configured to perform feature extraction on the first image set to obtain a first set of two-dimensional coordinate points and a first feature descriptor set associated with the first set of two-dimensional coordinate points; a second acquisition module configured to acquire map data, the map data comprising a set of three-dimensional coordinate points and a second feature descriptor set associated with the set of three-dimensional coordinate points, the first image set and the map data being for the same map; A first matching point pair set determination module is configured to determine, for a first image in the first image set, a first matching point pair set between a first subset of two-dimensional coordinate points associated with the first image and the set of three-dimensional coordinate points based on the first feature descriptor set and the second feature descriptor set; a first pose determination module, configured to determine a first pose of the first image relative to the set of three-dimensional coordinate points based on the first set of matching point pairs; A conversion module configured to convert the first subset of two-dimensional coordinate points into a first subset of three-dimensional coordinate points that can be used to expand the map data based on the first posture; A first image determination module is configured to determine a plurality of other images in the first image set that are spaced from the first image by a plurality of predetermined intervals; a second matching point pair determination module, configured to determine, based on the first feature descriptor set, a second matching point pair set between the first subset of two-dimensional coordinate points associated with the first image and multiple subsets of two-dimensional coordinate points associated with the multiple other images; as well as The first subset generation module is configured to generate a second subset of three-dimensional coordinate points that can be used to expand the map data based on the first matching point pair by triangulating the feature points if the image associated with any point in the first matching point pair in the second matching point pair set has a determined pose.

12. The device according to claim 11, wherein the first matching point pair set determination module comprises: A first feature descriptor determination module is configured to determine, for a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, based on a first feature descriptor in the first feature descriptor set that is associated with the first two-dimensional coordinate point and the second feature descriptor set, a second feature descriptor in the second feature descriptor set that is closest to the first feature descriptor; as well as The first matching point pair determination module is configured to determine the point pair associated with the first feature descriptor and the second feature descriptor as a matching point pair in the first matching point pair set if the distance between the first feature descriptor and the second feature descriptor is less than a first threshold distance.

13. The apparatus according to claim 11, wherein the first pose determination module comprises: A second pose determination module is configured to determine the first pose based on the first set of matching point pairs if the number of matching point pairs in the first set of matching point pairs associated with the first image is greater than the number of matching point pairs in each set of matching point pairs associated with each other image in the first image set whose pose has not yet been determined.

14. The device according to claim 11, wherein the second matching point pair determination module comprises: A second feature descriptor determination module is configured to determine, for a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, based on the first feature descriptor set, a third feature descriptor in the first subset of feature descriptors associated with a second image in the plurality of other images, the third feature descriptor being closest to the first feature descriptor associated with the first two-dimensional coordinate point; as well as The third matching point pair determination module is configured to determine the point pair associated with the first feature descriptor and the third feature descriptor as a matching point pair in the second matching point pair set if the distance between the first feature descriptor and the third feature descriptor is less than a second threshold distance.

15. The apparatus according to claim 11, further comprising: A subset determination module, configured to determine a subset of three-dimensional coordinate points in the first set of matching point pairs that are associated with the set of three-dimensional coordinate points; an image set determination module, configured to determine a second image set associated with the subset of the three-dimensional coordinate points for forming the map data; a fourth matching point pair determination module, configured to determine, based on the first feature descriptor set, a third matching point pair set between the first subset of two-dimensional coordinate points associated with the first image and the subset of two-dimensional coordinate points associated with the second image set; as well as The second subset generation module is configured to generate a third subset of three-dimensional coordinate points that can be used to expand the map data based on the third matching point pair set by triangulating the feature points.

16. The device according to claim 15, wherein the fourth matching point pair determination module comprises: A third feature descriptor determination module is configured to determine, for a first two-dimensional coordinate point in the first subset of two-dimensional coordinate points, based on the first feature descriptor set, a fourth feature descriptor in the second subset of feature descriptors associated with a third image in the second image set, the fourth feature descriptor being closest to the first feature descriptor associated with the first two-dimensional coordinate point; as well as The fifth matching point pair determination module is configured to determine the point pair associated with the first feature descriptor and the fourth feature descriptor as a matching point pair in the third matching point pair set if the distance between the first feature descriptor and the fourth feature descriptor is less than a third threshold distance.

17. The apparatus according to claim 11, wherein the first pose determination module comprises: A third posture determination module is configured to determine a second posture of the first image relative to the set of three-dimensional coordinate points based on the first set of matching point pairs; as well as The first optimization module is configured to optimize the second pose using a bundle adjustment nonlinear optimization algorithm based on a reprojection error constraint to obtain the first pose.

18. The apparatus of claim 17, wherein the first optimization module is further configured to optimize the second pose using the bundle adjustment nonlinear optimization algorithm based on at least one of: a pose constraint associated with two adjacent images in the first image set; a rotation angle constraint associated with the two adjacent images; and A point pair distance error constraint associated with the two adjacent images.

19. The apparatus according to claim 17, further comprising: A second optimization module is configured to optimize the first subset of three-dimensional coordinate points using the bundle adjustment nonlinear optimization algorithm based on the reprojection error constraint.

20. The apparatus according to claim 11, further comprising: a reprojection module, configured to reproject a first three-dimensional coordinate point in the first subset of three-dimensional coordinate points to obtain a reprojected two-dimensional coordinate point; as well as A deleting module is configured to delete the first three-dimensional coordinate point from the first subset of three-dimensional coordinate points if a distance between the reprojected two-dimensional coordinate point and a second two-dimensional coordinate point in the first subset of two-dimensional coordinate points and associated with the first three-dimensional coordinate point is greater than a fourth threshold distance.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Map merging method of unmanned aerial vehicle visual SLAM under city complex environment

    CN106595659A

  • Posture determination method, posture determination device, medium and computing device

    CN109087359A

  • Pose determination method and device based on depth information, medium and electronic equipment

    CN110349213A

  • Image processing method, device and system

    CN111213159A

  • Image processing method, apparatus, and storage medium

    US20190108646A1