Map construction method and device, electronic device, and storage medium

By performing feature extraction and matching relationship diagram generation on the two-dimensional image set, candidate image matching pairs are determined, and target matching pairs are determined through score evaluation of line feature candidate matching pairs, and accurate line maps are constructed, which solves the construction of line maps in the prior art. The problem of poor accuracy improves the accuracy of visual positioning and reduces hardware costs.

CN114419189BActive Publication Date: 2024-11-19GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210026040.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-11
Publication Date
2024-11-19
Estimated Expiration
2042-01-11

AI Technical Summary

Technical Problem

In the prior art, when building line maps, the lidar-based method has limitations, and the low-cost color camera hardware cannot be used directly, and the descriptor-based method has low matching accuracy, resulting in poor positioning accuracy.

Method used

By performing feature extraction and matching relationship diagram generation on the two-dimensional image set, point cloud data is constructed, candidate image matching pairs are determined, and target matching pairs are determined through score evaluation of line feature candidate matching pairs to build an accurate line map.

Benefits of technology

It improves the construction accuracy and application range of online maps, enhances the accuracy of visual positioning, reduces hardware costs, and is suitable for ordinary mobile phones and other devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114419189B_ABST
    Figure CN114419189B_ABST
Patent Text Reader

Abstract

The disclosed embodiments are about a map construction method and device, electronic device, and storage medium, and relate to the field of visual positioning technology. The map construction method includes: extracting features from images in a two-dimensional image set to generate a matching relationship graph, and reconstructing all images in the two-dimensional image set based on the matching relationship graph to construct point cloud data; extracting features from the image to be matched to obtain line features, determining candidate image matching pairs of the image to be matched based on the point cloud data, and determining line feature candidate matching pairs based on the candidate image matching pairs; performing score evaluation on the line feature candidate matching pairs to determine target matching pairs, and constructing a line map corresponding to the image to be matched based on the target matching pairs. The technical solution disclosed in the present invention can improve the accuracy of the constructed line map.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The present disclosure relates to the field of visual positioning technology, and in particular to a map construction method, a map construction device, an electronic device, and a computer-readable storage medium. Background Art

[0002] For visual positioning, the construction of point cloud maps and the camera pose solution based on point matching can meet most application scenarios, but the algorithm will fail for scenes with rich structured information but a large number of pure colors. Therefore, it is necessary to reconstruct line maps to assist visual positioning.

[0003] In the related art, a map is constructed based on line features of lidar data, or a line matching is performed directly using descriptors, thereby constructing a line feature map based on image data.

[0004] In the above methods, the application level of online feature maps is limited when relying on LiDAR and cannot be used directly. Using descriptors for matching has low matching precision and therefore poor accuracy.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a map construction method and device, an electronic device, and a storage medium, thereby overcoming the problem of being unable to accurately construct a line map generated due to limitations and defects of related technologies, at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a map construction method is provided, characterized in that it includes: performing feature extraction on images in a two-dimensional image set to generate a matching relationship graph, and reconstructing all images in the two-dimensional image set based on the matching relationship graph to construct point cloud data; performing feature extraction on images to be matched to obtain line features, determining candidate image matching pairs of the images to be matched based on the point cloud data, and determining line feature candidate matching pairs based on the candidate image matching pairs; performing score evaluation on the line feature candidate matching pairs to determine target matching pairs, and constructing a line map corresponding to the images to be matched based on the target matching pairs.

[0009] According to one aspect of the present disclosure, a map construction device is provided, including: a point cloud data construction module, used to perform feature extraction on images in a two-dimensional image set to generate a matching relationship graph, and reconstruct all images in the two-dimensional image set based on the matching relationship graph to construct point cloud data; a line feature matching module, used to perform feature extraction on images to be matched to obtain line features, determine candidate image matching pairs of the images to be matched according to the point cloud data, and determine line feature candidate matching pairs based on the candidate image matching pairs; a line map construction module, used to perform score evaluation on the line feature candidate matching pairs to determine target matching pairs, and construct a line map corresponding to the images to be matched according to the target matching pairs.

[0010] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the map construction methods described above by executing the executable instructions.

[0011] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the map construction method described in any one of the above is implemented.

[0012] In the map construction method, map construction device, electronic device and computer-readable storage medium provided in the embodiments of the present disclosure, on the one hand, candidate image matching pairs of the images to be matched are determined based on the point cloud data, line feature candidate matching pairs are determined based on the candidate image matching pairs, and the line feature candidate matching pairs are scored to determine the target matching pairs, and a line map is constructed based on the target matching pairs. Since line feature information is provided, limitations at the application level are avoided and the scope of application is increased. On the other hand, matching using line features and being able to determine the target matching pairs based on the similarity scores of the line feature candidate matching pairs can improve the accuracy of the matching pairs, thereby accurately constructing a line map and improving positioning accuracy.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0015] Figure 1A schematic diagram showing a system architecture to which the map construction method or map construction device according to the embodiments of the present disclosure can be applied.

[0016] Figure 2 A schematic diagram schematically illustrates a map construction method in an embodiment of the present disclosure.

[0017] Figure 3 The following is a schematic diagram of a process for determining an initial image matching pair in an embodiment of the present disclosure.

[0018] Figure 4 The schematic diagram of triangulation in the embodiment of the present disclosure is schematically shown.

[0019] Figure 5 The flowchart of reconstructing an image in an embodiment of the present disclosure is schematically shown.

[0020] Figure 6 The following is a schematic diagram showing the process of determining candidate matching pairs of line features in an embodiment of the present disclosure.

[0021] Figure 7 A schematic diagram schematically illustrates a candidate matching pair of line features in an embodiment of the present disclosure.

[0022] Figure 8 A schematic diagram of the line feature mapping result in an embodiment of the present disclosure is schematically shown.

[0023] Fig. 9 A schematic diagram schematically illustrates a sparse point cloud and a line map of a local area in an embodiment of the present disclosure.

[0024] Fig.10 A schematic diagram schematically illustrates the overall process of constructing a line map in an embodiment of the present disclosure.

[0025] Fig.11 A block diagram of a map construction device in an embodiment of the present disclosure is schematically shown.

[0026] Fig.12 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0028] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] The following methods are provided in the related technology: Method 1: Through an EKF-SLAM algorithm of a straight line feature map, laser radar is used to extract straight line features in the environment, and a line feature map is established based on the straight line features; the system performs radical regional segmentation and straight line fitting on the data measured by the laser radar to obtain the parameter equation of the straight line feature as the local line feature map in SLAM. Method 2: After the scanned laser data is subjected to noise reduction filtering, the laser point cloud data is segmented by the dynamic threshold method, and then the data in the area is fitted with line segments using the golden section method to obtain a local map; finally, the fan matching method is used to achieve the fusion of the local map to obtain a global map. Method 3: The essential matrix of adjacent images is estimated by the optical flow tracking method, and then the LSD line features are extracted, the LBD descriptor is calculated, and the descriptor is used for matching, and the intersection set of the midpoint polar line of each straight line in the reference frame and the matching straight line is calculated, and finally the intersection set is triangulated and straight line fitted, the reprojection error is constructed, the position of the line and the camera posture are optimized, and the line map is obtained.

[0030] In the above technical solutions, method one provides a local line map construction method for SLAM pose optimization, and method two fuses local maps to obtain a globally consistent line feature map. However, these two solutions mainly rely on the hardware equipment advantages of lidar, which only have structural features but no visual features. They have limitations in the application level of line feature maps and cannot be used directly on low-cost color camera hardware devices. Method three is the construction of line feature maps based on image data. This method directly uses LBD descriptors for straight line matching, and the matching accuracy is low. Wrong matching will cause errors in subsequent triangulation. In addition, the camera pose estimation of this method is optimized using the reprojection error of line features. Due to occlusion and other problems of line features under different viewing angles, the accuracy of the camera pose directly estimated by the reprojection error of the line is limited, which affects the final line map construction.

[0031] In order to solve the above technical problems, a map construction method is provided in an embodiment of the present disclosure, which can be applied to various types of augmented reality application scenarios, for example, it can be applied to application scenarios with less texture. Figure 1 A schematic diagram showing a system architecture to which the map construction method or map construction device according to the embodiments of the present disclosure can be applied.

[0032] like Figure 1 As shown, the system architecture 100 may include: a camera 110, a network 120, and an electronic device 130. The camera 110 refers to a camera configured with a single camera or a dual camera; the electronic device 130 may be a device with processing functions, such as a computer, a smart phone, a tablet computer, a smart wearable device (such as AR glasses), a robot, a drone, etc. The camera 110 may form a communication connection with the electronic device 130 through the network 120, and transmit the collected images or videos to the electronic device 130, which will analyze and process them. Figure 1 The camera 110 is shown to be disposed outside the electronic device 130. In one embodiment, the camera 110 may also be built in the electronic device 130, such as the electronic device 130 may be a smart phone or a robot equipped with a camera.

[0033] It should be noted that in this exemplary embodiment, a line map is constructed for the image captured by the camera 110 . If the camera 110 is built into the electronic device 130 , it is equivalent to constructing a line map for the image captured by the electronic device 130 .

[0034] It should be understood that Figure 1 The number of devices is only illustrative. For example, according to implementation needs, multiple cameras can be set up, which are respectively connected to the electronic device 130 through the network 120. The electronic device 130 can simultaneously construct a line map for the image of each camera, and so on.

[0035] Specifically, for the collected two-dimensional image set, point feature extraction and matching are first performed, and the SFM algorithm is used to restore the camera pose and the position of the feature point cloud to construct point cloud data. 2D line features are extracted on the image, and then the association between 3D points and images is used to generate the matching relationship between images. Finally, the epipolar geometry constraints are used to generate candidate matching pairs. The candidate matching pairs are triangulated, and the similarity of the 3D lines generated by each candidate matching pair is scored in three-dimensional space, and the 3D lines with higher scores are retained. Then the 3D lines are clustered, merged and simplified, and the 3D line map of the image to be reconstructed is output.

[0036] In order to solve the problems in the related art, a map construction method is provided in the embodiment of the present disclosure. Figure 2 The map construction method in the embodiment of the present disclosure is described in detail.

[0037] In step S210, feature extraction is performed on images in the two-dimensional image set to generate a matching relationship graph, and all images in the two-dimensional image set are reconstructed based on the matching relationship graph to construct point cloud data.

[0038] In the disclosed embodiment, a terminal can be used to collect a two-dimensional image set, and feature extraction and feature matching can be performed on the images in the two-dimensional image set to obtain a matching relationship graph. The terminal can be an intelligent terminal with a camera, or it can be other types of devices, as long as it can perform image acquisition. The features extracted here are point features of the image. Specifically, a learning point feature superpoint can be obtained, and the corresponding matching method superglue can be used to complete the matching process of the point features of the image. The matching relationship graph stores the feature points of all images in the two-dimensional image set, specifically including but not limited to the matching relationship between images and the matching relationship between feature points and feature points. Among them, the matching relationship between images is used to describe whether the images match, and the matching relationship between feature points and feature points is used to describe whether the feature points match. Based on this, information such as the number of feature point matching pairs and the number of image matching pairs can be obtained from the matching relationship graph.

[0039] While extracting point features, the two-dimensional image can also be reconstructed to obtain the camera pose and point cloud position. Specifically, the two-dimensional image can be reconstructed through SFM (Structure from Motion). Structure from Motion (SFM) uses a series of 2D images to obtain the 3D coordinates of the scene in the image for reconstruction. Structure from Motion is used to determine the spatial and geometric relationship of the target through the movement of the camera, and can include any of Euclidean structure recovery, affine structure recovery, and perspective structure recovery.

[0040] The camera pose is used to describe the posture of the camera. The 3D coordinates of the corresponding 2D points (point features) in the current camera coordinate system can be calculated first, and then the camera pose can be solved based on the 3D coordinates in the world coordinate system and the 3D coordinates in the current camera coordinate system. Point cloud data refers to the three-dimensional information of at least one pixel in an image. Through the coordinates of each pair of matching points, the coordinates of the matching points in space can be restored to obtain point cloud data. Among them, the point cloud data can be sparse point cloud data to save storage space.

[0041] When performing reconstruction, the reconstruction process can be a serialized reconstruction. The specific process of serialized reconstruction includes: first reconstructing the images of the two initial perspectives to obtain the corresponding initial point clouds and camera poses; further reconstructing the images of other perspectives to obtain reference three-dimensional points and reference poses; repeating this process until the images of all perspectives are successfully reconstructed. On this basis, the initial point cloud can be updated according to the reference three-dimensional points to obtain point cloud data (adding the reference three-dimensional points to the initial point cloud to generate point cloud data), and the camera pose can be updated according to the reference pose to obtain the final camera pose. The initial perspective can be the perspective that the user can see at the current moment, specifically any two perspectives. Other perspectives can be any perspective except the initial perspective.

[0042] Specifically, first, the image to be reconstructed in the current frame can be determined according to the feature point matching pair information in the matching relationship graph to determine the initial image matching pair, and the initial point cloud and camera pose are determined according to the initial image matching pair. Among them, the image to be reconstructed in the current frame refers to a frame of image to be reconstructed under the initial viewing angle. The feature point matching pair information refers to the total number of feature point matching pairs of each image. The feature point matching pairs are used to describe the feature points that match between each image and other images. The initial image matching pair refers to two images that are successfully matched under the initial viewing angle, for example, the current image and its associated image.

[0043] Figure 3 A flowchart for determining an initial image matching pair is schematically shown in FIG. Figure 3 As shown in , it mainly includes the following steps:

[0044] In step S310, all images are sorted according to the total number of feature point matching pairs on the images as the first image sequence of image matching pairs.

[0045] In this step, for each image, there may be an associated image with which it is associated to obtain an image matching pair. The image matching pair includes the first image and the second image. The two images in the image matching pair generate a feature point matching pair through matching feature points. The feature point matching pair information refers to the total number of feature point matching pairs corresponding to each image and all other associated images.

[0046] For example, from the matching relationship graph, each image is matched with its associated images to obtain an image matching pair. For example, image A and image B, image C, and image D all constitute an image matching pair. Each image matching pair will generate some feature point matching. For example, there are 500 feature point matching pairs between image A and image B, 400 feature point matching pairs between image A and image C, and 100 feature point matching pairs between image A and image D. The total number is the sum of all feature point matching pairs.

[0047] The first image sequence may be a sequence of all images to be reconstructed, which is used to represent the first image in the image matching pair, and all images in the first image sequence may be used as the current image. For example, the first image in the first image sequence is used as the current image, or the second image is used as the current image. Here, the first image is used as the current image as an example for explanation. Based on this, all images can be sorted in descending order according to the feature point matching pair information to determine the first image sequence of the image matching pair. That is, the frame with a large number of feature point matching pairs and rich texture is selected as the frame to be reconstructed, so that the multiple images selected are determined as the first image sequence.

[0048] In step S320, the first image sequence is traversed, and the second image sequence is determined according to the number of feature point matching pairs between the current image and other images in the first image sequence.

[0049] In this step, the second image sequence is used to represent multiple second images in the image matching pair, and the second image sequence can be a part of the first image sequence. Based on this, the first image sequence is traversed, and all images in the first image sequence are sorted in descending order according to the number of feature point matching pairs between the current image and other images to determine the second image sequence of the current image. Among them, the current image can be any one in the first image sequence, and the other images can be any one of the remaining images in the first image sequence except the current image. After the first image is determined, it is necessary to find the second image that best matches the first image to generate an initial image matching pair.

[0050] In step S330, the second image sequence is traversed, and if the current image and the associated image selected from other images satisfy the geometric constraint, the current image and the associated image are determined as an initial image matching pair.

[0051] In this step, the associated image refers to an image among other images that matches the current image, which can be any one in the second image sequence. Geometric constraints are used to describe the geometric relationship between different images, for example, two straight lines can be perpendicular or parallel, etc. If the associated image and the current image meet the set geometric constraints, the current image and the associated image can be determined as the initial image matching pair. For example, the initial image matching pair is image A and image B. If the geometric constraints are not met, the second image sequence is traversed until an associated image that meets the geometric constraints is obtained.

[0052] In the disclosed embodiment, after obtaining the initial image matching pair, the relative pose between the two images can be estimated based on the feature point matching relationship between the initial image matching pair (the matching relationship between feature points and feature points) to determine the camera pose. At the same time, the feature point matching pair can be triangulated to obtain an initial point cloud. The initial point cloud here refers to a sparse point cloud. Furthermore, the initial point cloud is optimized and filtered by constraints to improve the accuracy of the initial point cloud. The constraints may be epipolar constraints and viewing angle constraints. Among them, the epipolar constraint refers to a point-to-line constraint, rather than a point-to-point constraint, and the epipolar constraint is used to compress the corresponding point matching from searching for the entire image to searching for corresponding points on a straight line. Viewing angle constraints are used to describe the viewing angle range, for example, may include horizontal viewing angles or vertical viewing angles.

[0053] Next, the next frame can be reconstructed. The next frame can be used to represent other perspectives. Specifically, the target image in the two-dimensional image set is determined as the next frame of the image sequence to be reconstructed, and the image to be reconstructed in the next frame of the image sequence to be reconstructed is reconstructed according to the initial point cloud to generate reference three-dimensional points and reference poses. Among them, the target image can be determined according to the following conditions: (1) the frame image has not been reconstructed before; (2) the frame image has enough visible points in the existing 3D point cloud; (3) the number of attempts to reconstruct the frame image before does not exceed a certain fixed threshold, usually 3 times. The next frame of the image sequence to be reconstructed refers to a set of images that meet the next frame reconstruction conditions. The image to be reconstructed can be any one of the images, for example, it can be the first image in the next frame of the image sequence to be reconstructed, or it can also be the second image, etc. Here, the image to be reconstructed is taken as the first image as an example for explanation.

[0054] When reconstructing the image to be reconstructed in the next frame of the sequence of images to be reconstructed, first, a matching pair between the image to be reconstructed and the initial point cloud can be searched, that is, a 2D-3D matching pair between the image to be reconstructed and the existing three-dimensional point cloud (initial point cloud), that is, a matching pair between the image and the point cloud. Further, the PNP algorithm is used to calculate the reference pose of the image to be reconstructed in the initial point cloud. The PNP algorithm is used to recalculate the camera pose based on the known n spatial three-dimensional points (3D points) and the point pairs corresponding to the image 2D points on the basis of the camera pose. After determining the reference pose, the newly added image matching point pairs can be triangulated according to the reference pose to generate reference three-dimensional points. Reference three-dimensional points refer to three-dimensional points newly added due to triangulation.

[0055] Figure 4 The principle of triangulation is shown schematically in . Figure 4 As shown in , assume that the homogeneous coordinates of the three-dimensional space point P in the world coordinate system are X = [x, y, z, 1] T , the corresponding spatial point P's projection points in the two perspectives are p1 and p2 respectively, and its coordinates in their respective camera coordinate systems are: The camera projection matrices corresponding to the two images are P1 and P2, P1 = [P 11 ,P 12 ,P 13 ] T , P2=[P 21 ,P 22 ,P 23 ] T , where P 11 , P 12 , P 13 Corresponding to rows 1-3 of the projection matrix P1, P 21 , P 22 , P 23 Corresponding to rows 1-3 of the projection matrix P2, in an ideal situation, we have for By cross-multiplying it by itself on both sides, we get:

[0056]

[0057] Right now:

[0058]

[0059] You can get:

[0060]

[0061] Among them, formula (3) can be obtained by linear transformation of formula (1) and formula (2), so each perspective can provide two constraints. Combined with the second perspective, we can get: AX = 0, where:

[0062]

[0063] In formula (4), when the number of viewing points is small and there are no outliers, the matrix can be directly decomposed by SVD to obtain the coordinates of point P. When there are outliers, the RANSAC method can be used for estimation.

[0064] After obtaining the reference 3D points, the reference 3D points can be added to the initial point cloud, and the initial point cloud can be updated according to the reference 3D points to obtain point cloud data. This process is repeated until all images from all perspectives are reconstructed. And the reference 3D points reconstructed from each perspective can be added to the initial point cloud to obtain point cloud data.

[0065] It should be added that when the image to be reconstructed in the next frame of the sequence to be reconstructed is reconstructed successfully, the next frame of the sequence of images to be reconstructed is reselected according to the current point cloud and the reference pose for reconstruction to regenerate the reference three-dimensional points. When the image to be reconstructed in the next frame of the sequence to be reconstructed fails to be reconstructed, the next adjacent image of the image to be reconstructed is reconstructed as the image to be reconstructed to obtain the reference three-dimensional points until the reconstruction is successful. For example, the image to be reconstructed is the first image. When the first image in the next frame of the sequence to be reconstructed is successfully reconstructed, the next frame of the sequence of images to be reconstructed is reselected according to the current point cloud and the reference pose to continue the reconstruction. If the first image is not successfully reconstructed, the second image is reconstructed as the image to be reconstructed, and so on until the reconstruction is successful. Among them, the current point cloud refers to the point cloud data when the image in the next frame of the sequence to be reconstructed is successfully reconstructed.

[0066] After each frame of image is reconstructed, the current point cloud and camera pose of each successfully reconstructed frame of image can be locally optimized, and the camera pose and current point cloud of the successfully reconstructed image can be globally optimized when the successfully reconstructed image meets the reconstruction conditions, and finally a complete three-dimensional point cloud is obtained as point cloud data. The reconstruction condition refers to the increase of a fixed number of reconstructed images or the increase of a fixed number of reconstructed three-dimensional points. The optimization method of local optimization and global optimization can be BA (Bundle Adjustment). Bundle adjustment optimization is a process that can extract the coordinates of 3D points and the relative positions and optical information of each camera from these multi-view information.

[0067] Figure 5 The specific flow chart of image reconstruction is schematically shown in FIG. Figure 5As shown in , it mainly includes the following steps:

[0068] In step S501, an initial image matching pair is determined.

[0069] In step S502, the initial image matching pair is reconstructed.

[0070] In step S503, the next frame of the image sequence to be reconstructed is determined.

[0071] In step S504, the image to be reconstructed in the next frame of the sequence of images to be reconstructed is reconstructed. The image to be reconstructed may be the first image.

[0072] In step S505, it is determined whether the reconstruction is successful. If so, the process goes to step S506; if not, the process goes to step S504.

[0073] In step S506, local BA optimization is performed.

[0074] In step S507, it is determined whether the reconstruction frequency has been reached. The reconstruction frequency refers to the global BA frequency. If so, the process proceeds to step S508.

[0075] In step S508, global BA optimization is performed.

[0076] In step S509, it is determined whether all images have been reconstructed. If not, the process goes to step S503.

[0077] In the disclosed embodiment, the camera pose and initial point cloud are obtained by first reconstructing the current frame of the image to be reconstructed at the initial view angle, and then the next frame of the image to be reconstructed is reconstructed to obtain the current pose and reference three-dimensional points, and the reference three-dimensional points are added to the initial point cloud to supplement and update the initial point cloud to obtain point cloud data and realize serialized reconstruction. The global point map can be constructed through point features, reducing the inaccuracy caused by line feature pose estimation, estimating the accurate camera pose, and improving the accuracy of point cloud data.

[0078] Continue to refer Figure 2 As shown in , in step S220, feature extraction is performed on the image to be matched to obtain line features, candidate image matching pairs of the image to be matched are determined according to the point cloud data, and line feature candidate matching pairs are determined based on the candidate image matching pairs.

[0079] In the disclosed embodiment, the image to be matched can be any image in the two-dimensional image set. Candidate image matching pairs of the image to be matched can be determined based on the point cloud data. In line feature matching, image matching pairs are no longer determined by extracting image global descriptors and image retrieval, but are determined by the association relationship of the point cloud data generated in step S210. In the point cloud data, a 3D point is triangulated based on 2D points from various perspectives, and a 3D point will be seen by 2D points from many perspectives. Therefore, it can be determined through the point cloud data that there is an association relationship between the images where the 2D points are located.

[0080] Specifically, the three-dimensional feature points corresponding to some feature points on the image to be matched can be obtained according to the association relationship of the point cloud data; the image sets corresponding to multiple three-dimensional feature points are traversed, the number of elements in all image sets are sorted, and the candidate image matching pairs of the image to be matched are determined according to the sorting results. Specifically, first, there are M 2D feature points on the image A to be matched, of which N 2D feature points have corresponding 3D points, and each 3D point is obtained by triangulating several images including the image A to be matched. Therefore, some feature points are N 2D feature points, and three-dimensional feature points refer to 3D points, and three-dimensional feature points are the same as three-dimensional points. For the i-th 3D point, its corresponding image set is {I i0 ,I i1 ,...I ik}. Then, the image sets corresponding to the N 3D points are traversed respectively, the number of elements in each of the N sets is sorted from large to small, and the image I with the largest number of elements is determined as the image most relevant to the image A to be matched. On this basis, the candidate image matching pairs corresponding to the image A to be matched are selected from the N images according to the preset ratio and sorting order. The sorting order can be from large to small, and the preset ratio can be the first 20%.

[0081] After determining the candidate image matching pairs, the line feature candidate matching pairs can be calculated. Based on this, in the embodiment of the present disclosure, the image to be matched can first be feature extracted to obtain line features, that is, a line. Line features can be, for example, LCNN features. Usually, the matching of point features is measured by the distance of the descriptor, but line features are affected by factors such as perspective change and occlusion. The same line may have different lengths, and it is difficult for the descriptor to accurately express the characteristics of the line. Therefore, in the embodiment of the present disclosure, a geometric constraint method is used for line feature matching.

[0082] Figure 6 The flowchart for calculating the candidate matching pairs of line features is schematically shown in FIG. Figure 6 As shown in , it mainly includes the following steps:

[0083] In step S610, a basic matrix between the image to be matched and the matching image in the candidate image matching pair is calculated, and the endpoints of the reference line are determined according to the basic matrix and the endpoints of the line to be matched; the line to be matched is located in the image to be matched, and the reference line is located in the matching image;

[0084] In step S620, if the line to be matched and the reference line are determined to be matching line segments according to the overlap rate between the line to be matched and the reference line, the line to be matched and the reference line are determined to be a line feature candidate matching pair.

[0085] In the disclosed embodiment, the image to be matched and the matching image can form a candidate image matching pair. The basic matrix is ​​used to map between line features to map a line feature to other images.

[0086] Specifically, the basic matrix is ​​synthesized according to the transformation matrix of the relative world coordinate system between the image to be matched and the matching image. When determining the basic matrix, the basic matrix can be determined according to the rotation matrix, the translation vector, the internal parameter matrix and the essential matrix. Among them, the rotation matrices of the two images are R1 and R2, which are used to determine the rotation matrix in formula (5); the translation vectors are t1 and t2, which are specifically used to determine the translation vector of formula (6) and the translation matrix shown in formula (7); the internal parameter matrices are K1 and K2, then the essential matrix E is shown in formula (8), and the basic matrix F is shown in formula (9):

[0087]

[0088] t=t2-R*t1 Formula (6)

[0089]

[0090] E=T*R Formula (8)

[0091]

[0092] In the disclosed embodiment, the camera pose is obtained by SFM sparse point cloud reconstruction, and the rotation matrix and translation vector of each frame image in the world coordinate system have been restored, so the basic matrix between the two images can be directly synthesized by the transformation matrix of the two images relative to the world coordinate system. Compared with the traditional method of estimating the basic matrix of 2D feature points, the basic matrix obtained by reconstruction has more global optimal characteristics, introduces the constraints of 3D points and associated frames, and the basic matrix synthesized by camera pose is more accurate than the basic matrix estimated by 2D matching.

[0093] The line to be matched refers to the line feature of the image to be matched, so the line to be matched is located on the image to be matched, and the line to be matched can be any one of the line features. The reference line refers to the line feature of the matching image, so the reference line is located on the matching image, and can form a line feature candidate matching pair with the line to be matched. The endpoints of the line to be matched can be its left and right endpoints. Specifically, the image to be matched can be image I1, and the matching image is I2. The left and right endpoints of a line to be matched l1 on the image to be matched are respectively The polar equation e of the two endpoints on the matching image, image I2 left ,e right As shown in formula (10). A reference line l2 on image I2 can be defined by its left and right endpoints q left ,q right It is represented as shown in formula (11). At this time, formula (12) is used to transform the line to be matched l1 to the matching image I2, which can ensure that it is collinear with the reference line l2. Based on this, the coordinates of the left and right endpoints of the line to be matched on the matching image are Finally, by judging the two collinear line segment reference lines (q left ,q right ) and the line to be matched The overlap ratio of the two lines determines whether the two lines are matching line segments. The threshold of the overlap ratio may be, for example, 0.5. If the line to be matched and the reference line are matching line segments, the line to be matched and the reference line may be determined as a candidate matching pair of line features.

[0094]

[0095] l2=q left ×q right Formula (11)

[0096]

[0097] In step S230, the line feature candidate matching pairs are score evaluated to determine target matching pairs, and a line map corresponding to the image to be matched is constructed based on the target matching pairs.

[0098] In the disclosed embodiments, among the line matching pairs (i.e., line feature candidate matching pairs) obtained by means of geometric matching, there may be a one-to-many situation, that is, a line feature (line to be matched) on the image to be matched may have multiple reference lines on its corresponding matching image, that is, it is impossible to determine the line matching pairs one-to-one using the method of geometric constraints. In order to avoid this problem, a similarity score is constructed on the basis of matching, and a set of candidate matching pairs is returned as the target matching pairs. Specifically, multiple line feature candidate matching pairs may be triangulated, and the accuracy of the line feature candidate matching pairs may be measured in 3D space. Furthermore, a score evaluation may be performed on each line feature candidate matching pair, and the line feature candidate matching pair with the highest score may be determined as the target matching pair.

[0099] Specifically, the line feature candidate matching pair can first be triangulated. Line triangulation refers to triangulating the endpoints of the line segment represented by the line feature to obtain the endpoints in the three-dimensional space. By triangulating the endpoints in the three-dimensional space, the line segment in the three-dimensional space is determined.

[0100] After obtaining the line segments in the three-dimensional space, the scores of the line feature candidate matching pairs can be calculated, where the scores can be similarity scores. For a line to be matched, there can be multiple reference lines and their constituent line feature candidate matching pairs. Therefore, for each current line feature candidate matching pair, the similarity score of each current line feature candidate matching pair can be calculated by the similarity score between the current line feature candidate matching pair and other line feature matching pairs. Specifically, the similarity score can be calculated based on the angle similarity and the distance similarity. Among them, formula (13) is used to represent the angle, the angle similarity can be calculated based on formula (14), and the distance similarity can be calculated based on formula (15) and formula (16):

[0101] angle=max(min(l1·l2,-1.0),1.0)*PI / 180 Formula (13)

[0102] similarity_angle=exp(-angle*angle / σ a ) Formula (14)

[0103] pos=d1-d2 Formula (15)

[0104] similarity_pos=exp(-d1*d1 / σ p ) Formula (16)

[0105] Among them, d1 and d2 are the midpoint depth values ​​of l1 and l2 respectively, σ a ,σ p are the regularization parameters for angle similarity and distance similarity, respectively.

[0106] After calculating the similarity score of each line feature candidate matching pair, the line feature candidate matching pair with the highest score is selected as the most accurate matching pair, and its corresponding three-dimensional line segment is retained. In the embodiment of the present disclosure, the angle similarity and the distance similarity are first converted to the same scale range according to the regularization parameter, and the angle similarity and the distance similarity are combined to obtain the similarity score. Figure 7 As shown in the figure, the matching images of the image A to be matched are B, C, and D. The line segment a on the image A to be matched can find 5 groups of line feature candidate matching pairs, namely ab, ac, ad, ae, and af, on the images B, C, and D. Among them, for the line feature candidate matching pair ab, its similarity score can be determined according to the sum of the similarity scores of the 3D line segments generated by the line feature candidate matching pair ab and the 3D line segments generated by the other line feature candidate matching pairs. If the similarity score corresponding to the line feature candidate matching pair ab is the highest, ab is taken as the target matching pair.

[0107] After determining the target matching pair, if the similarity score of the target matching pair is greater than the threshold, it can be considered that the two line segments in a set of three-dimensional line segments corresponding to the target matching pair belong to the same three-dimensional line segment. Based on this, the line segments to be matched and the reference line segments corresponding to the target matching pair can be clustered and fused to obtain the target line segment. For example, if the similarity score of the target matching pair ab is greater than the threshold, it can be considered that the line segments to be matched a and the reference line segments b contained therein belong to the same three-dimensional line segment, so the line segments to be matched a and the reference line segments b can be clustered and fused to obtain the target line segment. Based on this, a line map of the image to be matched can be constructed according to the target line segment to simplify the reconstructed line map.

[0108] Figure 8 A 3D line map of a reconstructed data set of an object 1 is schematically shown. Figure 8 Figure A in the figure is a large scene line map. For a clearer display, Figure 8 Figure B shows the local area. Figure 8 Figure C in FIG. 1 is the corresponding two-dimensional image. Fig. 9 is the corresponding sparse point cloud and line map. Fig. 9 It can be seen that in areas where point features are not rich, line features are relatively rich, which can provide more observation information for visual positioning.

[0109] Fig.10 The flowchart for constructing line features is shown schematically in Fig.10 As shown in , it mainly includes the following steps:

[0110] In step S1010, an image is acquired.

[0111] In step S1020, a sparse point cloud is constructed for the image, which includes: step S1021, point feature extraction and matching; step S1022, pose estimation; step S1023, triangulation; step S1024, bundle adjustment optimization.

[0112] In step S1030, candidate image matching pairs of the images to be matched are determined, which includes the following steps: step S1031, line feature extraction; step S1032, epipolar geometry calculation; step S1033, line feature matching.

[0113] In step S1040, three-dimensional line segments are constructed, which includes the following steps: step S1041, line triangulation; step S1042, line segment scoring and filtering; step S1043, clustering optimization.

[0114] In step S1050, a globally consistent line map corresponding to the image to be matched is obtained.

[0115] In the disclosed embodiment, a method for constructing a line feature map based on a sparse visual point cloud is provided. First, a global point map is constructed through point features to estimate the accurate camera pose and reduce the inaccuracy caused by the line feature pose estimation. Then, on the basis of obtaining the global camera pose, line feature matching is performed using geometric relationships instead of using line descriptors such as LBD, and according to the 3D line segment scoring mechanism, the correct line matching pairs and corresponding 3D line segments are selected to the greatest extent, and the 3D line segments in the same line cluster are clustered and optimized to obtain a globally consistent three-dimensional line feature map, thereby improving the accuracy and consistency of the line map. Compared with laser radar, the hardware cost is reduced, and it can be directly applied on ordinary mobile phones, cameras and other devices, thereby increasing the scope of the application layer, making it easier to implement and increasing convenience. In addition, the use of line features to assist in positioning can greatly improve the success rate and accuracy of positioning.

[0116] The disclosed embodiments can be applied to applications such as AR navigation and AR games based on visual positioning technology, and can also be applied to the external parameter calibration of visual sensors and laser sensors. The premise of visual positioning is to obtain a three-dimensional map of the scene and obtain a mapping from the physical world to the digital world. For scenes with weak textures but rich structured information, by ignoring the features at the pixel level, only geometric relationships are used to match 2D line segments, and line feature candidate matching pairs are filtered in three-dimensional space. The three-dimensional line segments triangulated from each line feature candidate matching pair are measured for similarity, and only a group of matches with higher similarity are retained as target matching pairs, thereby filtering out erroneous matches and reconstructing an accurate 3D line feature map through double matching. Line feature matching through geometric methods improves accuracy compared to traditional descriptor methods. Using line features to assist positioning can greatly improve the success rate and accuracy of positioning.

[0117] In an application environment with less texture, it is difficult to construct a rich point cloud map, resulting in positioning failure. In the disclosed embodiment, structured line features are used to construct a visual line feature map to assist in visual positioning. For example, in a conference room environment, there are only solid-color objects with obvious geometric structures such as tables, chairs, and screens. The technical solution in the disclosed embodiment can use structured line features to construct a visual line feature map to assist in visual positioning. Thereby, AR effects such as AR conferences and other functions are realized in the conference room environment, improving the AR experience and thus increasing the scope of application.

[0118] In the embodiment of the present disclosure, a map construction device is also provided. Fig.11 As shown in , the map construction device 1100 specifically includes:

[0119] The point cloud data construction module 1101 is used to extract features from images in the two-dimensional image set to generate a matching relationship graph, and reconstruct all images in the two-dimensional image set based on the matching relationship graph to construct point cloud data;

[0120] The line feature matching module 1102 is used to extract features from the image to be matched to obtain line features, determine candidate image matching pairs of the image to be matched according to the point cloud data, and determine candidate line feature matching pairs based on the candidate image matching pairs;

[0121] The line map construction module 1103 is used to perform score evaluation on the candidate matching pairs of line features to determine target matching pairs, and to construct a line map corresponding to the image to be matched according to the target matching pairs.

[0122] In an exemplary embodiment of the present disclosure, the point cloud data construction module includes: an initial matching module, which is used to determine the current frame of the image to be reconstructed from the two-dimensional image set to determine the initial image matching pair according to the feature point matching pair information in the matching relationship diagram, and determine the initial point cloud and the camera pose according to the initial image matching pair; a next frame matching module, which is used to determine the next frame of the image sequence to be reconstructed in the two-dimensional image set, reconstruct the image to be reconstructed in the next frame of the image sequence to be reconstructed according to the initial point cloud, generate a reference pose and obtain a reference three-dimensional point based on the reference pose; a point cloud update module, which is used to update the initial point cloud according to the reference three-dimensional point pair to obtain point cloud data.

[0123] In an exemplary embodiment of the present disclosure, the initial matching module includes: a first image determination module, which is used to sort all images according to the total number of feature point matching pairs on the images as a first image sequence of image matching pairs; a second image determination module, which is used to traverse the first image sequence and determine the second image sequence according to the number of feature point matching pairs between a current image and other images in the first image sequence; an initial matching determination module, which is used to traverse the second image sequence and determine the current image and the associated image selected from other images as an initial image matching pair if the current image and the associated image meet geometric constraints.

[0124] In an exemplary embodiment of the present disclosure, the initial matching module includes: a posture determination module, which is used to estimate the relative posture between the initial image matching pairs based on the feature point matching relationship between the initial image matching pairs to determine the camera posture; an initial point cloud determination module, which is used to triangulate the feature point matching pairs of the initial image matching pairs to obtain an initial point cloud.

[0125] In an exemplary embodiment of the present disclosure, the next frame matching module includes: a matching pair module, used to search for matching pairs between the image to be reconstructed and the initial point cloud; a reference pose determination module, used to calculate the reference pose of the image to be reconstructed in the initial point cloud based on the matching pairs; a reference three-dimensional point determination module, used to triangulate the newly added image matching point pairs based on the reference pose to generate reference three-dimensional points.

[0126] In an exemplary embodiment of the present disclosure, the device also includes: a reconstruction failure module, which is used to re-select the next frame of the image sequence to be reconstructed for reconstruction according to the current point cloud and the reference pose when the reconstruction of the image to be reconstructed in the next frame of the sequence to be reconstructed is successful, so as to regenerate the reference three-dimensional points; a reconstruction success module, which is used to reconstruct the next adjacent image of the image to be reconstructed as the image to be reconstructed when the reconstruction of the image to be reconstructed in the next frame of the sequence to be reconstructed fails, until the reconstruction is successful.

[0127] In an exemplary embodiment of the present disclosure, the device also includes: an optimization module, which is used to locally optimize the camera pose and current point cloud corresponding to each successfully reconstructed image, and to globally optimize the camera pose and current point cloud corresponding to all successfully reconstructed images to obtain the point cloud data; all the successfully reconstructed images meet the reconstruction conditions.

[0128] In an exemplary embodiment of the present disclosure, the line feature matching module includes: a three-dimensional feature point acquisition module, which is used to obtain multiple three-dimensional feature points corresponding to some feature points on the image to be matched according to the association relationship of the point cloud data; a candidate image matching pair determination module, which is used to traverse the image set corresponding to the multiple three-dimensional feature points, sort the number of elements in all image sets, and determine the candidate image matching pairs of the image to be matched according to the sorting result.

[0129] In an exemplary embodiment of the present disclosure, a line feature matching module includes: an endpoint determination module, which is used to calculate a basic matrix between an image to be matched and a matching image in a candidate image matching pair, and determine the endpoints of a reference line based on the basic matrix and the endpoints of the line to be matched; the line to be matched is located in the image to be matched, and the reference line is located in the matching image; a matching judgment module, which is used to determine that the line to be matched and the reference line are a matching line segment if the overlap rate between the line to be matched and the reference line determines that the line to be matched and the reference line are a line feature candidate matching pair.

[0130] In an exemplary embodiment of the present disclosure, the basic matrix calculation module includes: a synthesis module, which is used to synthesize the basic matrix according to the transformation matrix of the relative world coordinate system between the image to be matched and the matching image.

[0131] In an exemplary embodiment of the present disclosure, the device also includes: a co-linear module, which is used to convert the line to be matched on the image to be matched to the matching image so that the line to be matched and the reference line are co-linear, and determine the matching line segment according to the overlap rate of the co-linear line to be matched and the reference line.

[0132] In an exemplary embodiment of the present disclosure, the line map construction module includes: a target matching pair determination module, which is used to triangulate the line feature candidate matching pairs and calculate the similarity score of each line feature candidate matching pair, and take the line feature candidate matching pair with the largest similarity score as the target matching pair.

[0133] In an exemplary embodiment of the present disclosure, the target matching pair determination module includes: a similarity score calculation module, which is used to calculate the similarity score of each current line feature candidate matching pair through the similarity scores of the three-dimensional line segments of the current line feature candidate matching pair and the three-dimensional line segments of other line feature matching pairs.

[0134] It should be noted that the specific details of each module in the above-mentioned map construction device have been described in detail in the corresponding map construction method, so they will not be repeated here.

[0135] Fig.12 Schematic diagram of an electronic device suitable for implementing an exemplary embodiment of the present disclosure is shown. The terminal of the present disclosure may be configured as follows Fig.12The form of the electronic device shown, however, needs to be explained. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0136] The electronic device of the present disclosure includes at least a processor and a memory, wherein the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the method of the exemplary embodiment of the present disclosure.

[0137] Specifically, Fig.12 As shown, the electronic device 1200 may include: a processor 1210, an internal memory 1221, an external memory interface 1222, a Universal Serial Bus (USB) interface 1230, a charging management module 1240, a power management module 1241, a battery 1242, an antenna 1, an antenna 2, a mobile communication module 1250, a wireless communication module 1260, an audio module 1270, a speaker 1271, a receiver 1272, a microphone 1273, an earphone interface 1274, a sensor module 1280, a display screen 1290, a camera module 1291, an indicator 1292, a motor 1293, a button 1294 and a Subscriber Identification Module (SIM) card interface 1295, etc. The sensor module 1280 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor and a bone conduction sensor, etc.

[0138] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1200. In other embodiments of the present application, the electronic device 1200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0139] The processor 1210 may include one or more processing units, for example: the processor 1210 may include an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor, a baseband processor and / or a neural network processor (Neural-etwork Processing Unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. In addition, a memory can also be set in the processor 1210 for storing instructions and data. The map construction method in this exemplary embodiment can be executed by an application processor, a graphics processor or an image signal processor. When the method involves neural network-related processing, it can be executed by the NPU.

[0140] The internal memory 1221 can be used to store computer executable program codes, which include instructions. The internal memory 1221 can include a program storage area and a data storage area. The external memory interface 1222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1200.

[0141] The communication function of the mobile terminal 1200 can be implemented by a mobile communication module, an antenna 1, a wireless communication module, an antenna 2, a modulation and demodulation processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1400. The wireless communication module can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 200.

[0142] The display screen is used to implement display functions, such as displaying user interfaces, images, and videos. The camera module is used to implement shooting functions, such as shooting images and videos. The audio module is used to implement audio functions, such as playing audio and collecting voice. The power module is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status.

[0143] The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0144] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0145] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0146] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0147] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0148] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0149] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0150] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims. It should be understood that the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A map construction method, characterized in that: include: Extracting features from images in the two-dimensional image set to generate a matching relationship graph, and reconstructing all images in the two-dimensional image set based on the matching relationship graph to construct point cloud data; Extracting features of the image to be matched to obtain line features, determining candidate image matching pairs of the image to be matched according to the point cloud data, and determining candidate line feature matching pairs based on the candidate image matching pairs; Score evaluation is performed on the candidate line feature matching pairs, a target matching pair is determined according to the similarity scores of the candidate line feature matching pairs, and a line map corresponding to the image to be matched is constructed according to the target matching pairs; Wherein, determining the candidate image matching pairs of the image to be matched according to the point cloud data includes: According to the association relationship of the point cloud data, multiple three-dimensional feature points corresponding to some feature points on the image to be matched are obtained; the image sets corresponding to the multiple three-dimensional feature points are traversed, the number of elements in all image sets is sorted, and the candidate image matching pairs of the image to be matched are determined according to the sorting result; Determining a line feature candidate matching pair based on the candidate image matching pair includes: Calculate the basic matrix between the image to be matched and the matching image in the candidate image matching pair, and determine the endpoints of the reference line in the matching image based on the basic matrix and the endpoints of the line to be matched located in the image to be matched; if it is determined that the line to be matched and the reference line are matching line segments, determine that the line to be matched and the reference line are a line feature candidate matching pair.

2. The map construction method according to claim 1, characterized in that: The reconstructing all images in the two-dimensional image set based on the matching relationship graph to construct point cloud data includes: According to the feature point matching pair information in the matching relationship graph, determine the current frame to be reconstructed from the two-dimensional image set to determine the initial image matching pair, and determine the initial point cloud and camera pose according to the initial image matching pair; Determine a next frame of an image sequence to be reconstructed in the two-dimensional image set, reconstruct the image to be reconstructed in the next frame of the image sequence to be reconstructed according to the initial point cloud, generate a reference pose, and obtain a reference three-dimensional point based on the reference pose; The initial point cloud is updated according to the reference 3D points to obtain point cloud data.

3. The map construction method according to claim 2, characterized in that: The step of determining the current frame to be reconstructed image from the two-dimensional image set according to the feature point matching pairs in the matching relationship graph to determine the initial image matching pairs comprises: Sort all images according to the total number of feature point matching pairs on the images as the first image sequence of image matching pairs; Traversing the first image sequence, and determining the second image sequence according to the number of feature point matching pairs between the current image and other images in the first image sequence; The second image sequence is traversed, and if the current image and the associated image selected from other images satisfy the geometric constraint, the current image and the associated image are determined as an initial image matching pair.

4. The map construction method according to claim 2, characterized in that: The determining of the initial point cloud and the camera pose according to the initial image matching pair comprises: According to the feature point matching relationship between the initial image matching pairs, the relative pose between the initial image matching pairs is estimated to determine the camera pose; The feature point matching pairs of the initial image matching pairs are triangulated to obtain the initial point cloud.

5. The map construction method according to claim 2, characterized in that: The step of reconstructing the image to be reconstructed in the next frame of the image sequence to be reconstructed according to the initial point cloud, generating a reference pose, and obtaining a reference three-dimensional point based on the reference pose includes: Searching for a matching pair between the image to be reconstructed and the initial point cloud; Calculate the reference pose of the image to be reconstructed in the initial point cloud according to the matching pair; Based on the reference pose, the newly added image matching point pairs are triangulated to generate reference three-dimensional points.

6. The map construction method according to claim 2 or 5, characterized in that: The method further comprises: When the image to be reconstructed in the next frame of the sequence to be reconstructed is reconstructed successfully, the next frame of the sequence to be reconstructed is reselected according to the current point cloud and the reference pose to regenerate the reference 3D points; When the reconstruction of the image to be reconstructed in the next frame of the sequence to be reconstructed fails, the next adjacent image of the image to be reconstructed is reconstructed as the image to be reconstructed again for reconstruction until the reconstruction succeeds.

7. The map construction method according to claim 2, characterized in that: The method further comprises: The camera pose and current point cloud corresponding to each successfully reconstructed image are locally optimized, and the camera pose and current point cloud corresponding to all successfully reconstructed images are globally optimized to obtain the point cloud data; all successfully reconstructed images meet the reconstruction conditions.

8. The map construction method according to claim 1, characterized in that: The determining of the line feature candidate matching pairs based on the candidate image matching pairs comprises: If the line to be matched and the reference line are determined to be a matching line segment according to the overlap rate between the line to be matched and the reference line, the line to be matched and the reference line are determined to be a line feature candidate matching pair.

9. The map construction method according to claim 1, characterized in that: The calculating of the basic matrix between two matching images in the candidate image matching pair includes: The basic matrix is ​​synthesized according to the transformation matrix of the relative world coordinate system between the image to be matched and the matching image.

10. The map construction method according to claim 8, characterized in that: The method further comprises: The line to be matched on the image to be matched is converted to the matching image so that the line to be matched and the reference line are collinear, and the matching line segment is determined according to the overlap rate of the collinear line to be matched and the reference line.

11. The map construction method according to claim 1, characterized in that: The step of performing score evaluation on the candidate matching pairs and determining a target matching pair according to the similarity scores of the line feature candidate matching pairs includes: The line feature candidate matching pairs are triangulated and the similarity score of each line feature candidate matching pair is calculated, and the line feature candidate matching pair with the largest similarity score is taken as the target matching pair.

12. The map construction method according to claim 11, characterized in that: The calculating of the similarity score of each line feature candidate matching pair includes: The similarity score of each current line feature candidate matching pair is calculated by comparing the three-dimensional line segments of the current line feature candidate matching pair with the three-dimensional line segments of other line feature matching pairs.

13. A map construction device, characterized in that: include: A point cloud data construction module is used to extract features from images in a two-dimensional image set to generate a matching relationship graph, and reconstruct all images in the two-dimensional image set based on the matching relationship graph to construct point cloud data; A line feature matching module is used to extract features from the image to be matched to obtain line features, determine candidate image matching pairs of the image to be matched according to the point cloud data, and determine candidate line feature matching pairs based on the candidate image matching pairs; A line map construction module, used to score the line feature candidate matching pairs, determine the target matching pairs according to the similarity scores of the line feature candidate matching pairs, and construct a line map corresponding to the image to be matched according to the target matching pairs; Wherein, determining the candidate image matching pairs of the image to be matched according to the point cloud data includes: According to the association relationship of the point cloud data, multiple three-dimensional feature points corresponding to some feature points on the image to be matched are obtained; the image sets corresponding to the multiple three-dimensional feature points are traversed, the number of elements in all image sets is sorted, and the candidate image matching pairs of the image to be matched are determined according to the sorting result; Determining a line feature candidate matching pair based on the candidate image matching pair includes: Calculate the basic matrix between the image to be matched and the matching image in the candidate image matching pair, and determine the endpoints of the reference line in the matching image based on the basic matrix and the endpoints of the line to be matched located in the image to be matched; if it is determined that the line to be matched and the reference line are matching line segments, determine that the line to be matched and the reference line are a line feature candidate matching pair.

14. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the map construction method described in any one of claims 1-12 by executing the executable instructions.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the map construction method according to any one of claims 1 to 12 is implemented.

Citation Information

Patent Citations

  • Dotted line based robot initial posture acquisition method and start-up self-positioning method

    CN111283730A

  • A method and device for determining pose of visual device

    CN111325796A

  • Map data updating method and device, storage medium and electronic equipment

    CN112269851A

  • Map construction method and device and electronic equipment

    CN113465617A

  • Computer-based systems configured for automated feature detection for image analysis and methods thereof

    WO2021155082A1