Method, device and equipment for updating three-dimensional map constructed based on panorama

Abstract

The invention provides a method, a device and equipment for updating a three-dimensional map constructed based on a panorama, and belongs to the technical field of computer vision and three-dimensional reconstruction. The method comprises the following steps: acquiring a to-be-positioned image, and calling a feature extraction model to extract image feature information from the to-be-positioned image; calling an image matching model to query a to-be-updated image which is most matched with the image feature information from a to-be-updated three-dimensional map; identifying a first pose of the to-be-positioned image in the to-be-updated three-dimensional map based on the to-be-updated image; calculating point cloud data of the to-be-positioned image based on the first pose; and updating the to-be-positioned image, the first pose and the point cloud data to the to-be-updated three-dimensional map. According to the invention, the updating flexibility and updating efficiency of the three-dimensional map can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and three-dimensional reconstruction, and in particular to a method, device and equipment for updating a three-dimensional map constructed based on a panoramic image. Background Art

[0002] In the prior art, Mixed Reality (MR), Augmented Reality (AR) and Virtual Reality (VR) technologies are widely used in entertainment, education, medical and other fields due to their highly immersive experience. The core of these technologies is to create a virtual environment that can interact with the real world or completely simulate a new world. In order to build such a virtual environment, panoramic cameras are widely used to capture the surrounding environment in all directions and generate three-dimensional maps based on these image data. For example, patent CN118918273A discloses in detail how to obtain environmental images through panoramic cameras and match them with map information in the database of a remote cloud server to achieve three-dimensional reconstruction of the scene environment.

[0003] However, in the real world, many environments are changing dynamically, and pre-made maps often cannot reflect these changes in a timely manner. For example, factors such as the demolition of buildings, the reconstruction of roads, and the rearrangement of furniture may affect the positioning accuracy of the device; in addition, pre-made maps are difficult to cover all spatial environments at once, resulting in a lack of map information in some areas.

[0004] Traditional map updating methods usually require manual intervention, which is not only inflexible but also inefficient. Therefore, it is necessary to propose a new map updating method to improve the flexibility and efficiency of map updating. Summary of the invention

[0005] The object of the present invention is to provide a method, device and equipment for updating a three-dimensional map constructed based on a panoramic image, so as to improve the flexibility and updating efficiency of the map.

[0006] In a first aspect of the present application, a method for updating a three-dimensional map constructed based on a panoramic image is provided, the method comprising: Acquire an image to be located, and call a feature extraction model to extract image feature information from the image to be located; Calling the image matching model to query the image to be updated that best matches the image feature information from the three-dimensional map to be updated; Identify the first pose of the image to be located in the three-dimensional map to be updated based on the image to be updated; Calculating point cloud data of the image to be positioned based on the first posture; The image to be located, the first pose and the point cloud data are updated into the three-dimensional map to be updated.

[0007] Optionally, the image matching model includes an image retrieval model and a feature matching model, and calling the image matching model to query the image to be updated that best matches the image feature information from the three-dimensional map to be updated includes: Calling the image retrieval model to query a first number of candidate images matching the image feature information from the three-dimensional map to be updated; Calling the image matching model to calculate a set of feature point matching pairs between the image to be located and each candidate image; An image to be updated is determined from the candidate images based on the feature point matching pair set.

[0008] Optionally, the calling the image matching model to calculate a set of feature point matching pairs between the image to be located and each candidate image includes: Identifying a scene of the image to be located, wherein the scene includes an indoor scene and an outdoor scene; An image matching model matching the scene is called to calculate a set of feature point matching pairs between the image to be located and each candidate image.

[0009] Optionally, the step of identifying a first pose of the image to be located in the three-dimensional map to be updated based on the image to be updated includes: Based on the feature point matching pair set between the image to be positioned and the image to be updated, a preset random sample consensus algorithm is called to calculate an inner point matching pair set; and the first posture is calculated based on the inner point matching pair set.

[0010] Optionally, before updating the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated, the method further includes: Determining whether the image to be located satisfies a first update condition based on a first number of point pairs in a feature point matching pair set between the image to be located and the image to be updated and / or a second number of point pairs in the interior point matching pair set; When the first update condition is met, the step of updating the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated is performed.

[0011] Optionally, before updating the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated, the method further comprises: calculating the difference between the first pose and an existing pose in the three-dimensional map to be updated; Determining whether the image to be located satisfies a second update condition based on the difference; When the second update condition is met, the image to be located, the first pose and the point cloud data are updated into the three-dimensional map to be updated.

[0012] Optionally, the second update condition includes that the difference exceeds a preset first difference threshold; The updating of the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated comprises: When the second update condition is met, adding the image to be located, the first pose and the point cloud data to the three-dimensional map to be updated; When the second update condition is not met and the posture difference between the image to be updated and the image to be positioned is less than a preset second difference threshold, the information of the image to be updated in the three-dimensional map to be updated is replaced by the image to be positioned, the first posture and the point cloud data to achieve the update of the three-dimensional map to be updated, and the second difference threshold is less than the first difference threshold.

[0013] Optionally, the feature extraction model includes a global feature extraction model and a local feature extraction model, and the image feature information includes a global feature vector and a local feature vector; The calling of the feature extraction model to extract image feature information from the image to be located includes: Calling a global feature extraction model to extract a global feature vector from the image to be located; A local feature extraction model is called to extract a local feature vector from the image to be located.

[0014] In a second aspect of the present application, a device for updating a three-dimensional map constructed based on a panoramic image is provided, the device comprising: A feature extraction module is used to obtain an image to be located and call a feature extraction model to extract image feature information from the image to be located; An image matching module, used for invoking an image matching model to query an image to be updated that best matches the image feature information from the three-dimensional map to be updated; A posture calculation module, used for identifying the first posture of the image to be located in the three-dimensional map to be updated based on the image to be updated; A map updating module is used to calculate the point cloud data of the image to be located based on the first posture, and update the image to be located, the first posture and the point cloud data into the three-dimensional map to be updated.

[0015] In a third aspect of the present application, an electronic device is provided, including: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to execute the method for updating a three-dimensional map constructed based on a panoramic image as described in any one of the embodiments of the present application.

[0016] The updating method, device and equipment of the three-dimensional map constructed based on the panoramic image in the present application can improve the accuracy of the first pose calculation by extracting image feature information of the image to be located, combining the image feature information of the image in the existing three-dimensional map, querying the image to be updated in the three-dimensional map that best matches the image to be located, and combining the queried image to be updated to calculate the first pose of the image to be located, thereby realizing high-precision real-time positioning of the image to be located, and avoiding the problem of inaccurate positioning when the position of the image to be located is too far from the three-dimensional reconstruction scanning trajectory.

[0017] Furthermore, under the premise of ensuring the accuracy of the first pose calculation, the point cloud data of the image to be located is calculated based on the first pose, and then the three-dimensional image is updated based on the point cloud data, thereby continuously optimizing and improving the accuracy and real-time performance of the three-dimensional map, and improving the real-time performance and update quality of the three-dimensional image update. In addition, the present application can realize automatic three-dimensional map updates without the need for manual user operation updates, thereby improving the flexibility and update efficiency of map updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope of the present application.

[0019] Figure 1 A schematic diagram of a flow chart of a method for updating a three-dimensional map constructed based on a panoramic image in one embodiment; Figure 2 A schematic diagram of a process of calling an image matching model to query an image to be updated that best matches the image feature information from a three-dimensional map to be updated in one embodiment; Figure 3 A schematic flow chart of a method for updating a three-dimensional map constructed based on a panoramic image in another embodiment; Figure 4 It is a structural block diagram of a device for updating a three-dimensional map constructed based on a panoramic image in one embodiment; Figure 5 It is a structural block diagram of a device for updating a three-dimensional map constructed based on a panoramic image in another embodiment; Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] All terms (including technical and scientific terms) used in this application have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0022] For example, the terms "first", "second", etc. used in this application are only used to distinguish similar objects and to differentiate the first object from another object, rather than to describe a specific order or sequence, and cannot be understood as indicating or implying relative importance.

[0023] This application proposes a method for updating a three-dimensional map based on a panoramic image. Figure 1 As shown, the method includes: Step 110: obtain the image to be positioned, and call the feature extraction model to extract image feature information from the image to be positioned.

[0024] In this embodiment, the image to be located is an image of the scene corresponding to the three-dimensional map captured by the end-side device, and the image may be a panoramic image or a normal perspective image. The end-side device may be any terminal with a shooting function, such as an intelligent robot, a user's AR glasses, a mobile phone, a camera, a drone, etc. The three-dimensional map may be a three-dimensional map pre-constructed based on a panoramic map, and the construction method of the three-dimensional map may be a scene map constructed by any suitable three-dimensional reconstruction method. The scenes involved in the three-dimensional map may include one or more of indoor scenes and / or outdoor scenes, such as exhibition space scenes, scenes suitable for AR glasses positioning and roaming, etc. Specifically, by collecting environmental data, using a panoramic camera to capture images, and generating a 3D map through a preset visual algorithm, the visual algorithm may be, for example, a SLAM algorithm. For example, the three-dimensional map is a three-dimensional map in an immersive experience scene such as MR, VR, and AR constructed according to the method described in patents such as CN118918273A or CN118840482A.

[0025] For example, when a user is in a scene corresponding to the three-dimensional map (such as an immersive experience scene such as MR, VR, AR, etc.), he can use his mobile phone to take one or more images of the current environment and upload the images to an electronic device. The electronic device can be a cloud server or any other suitable device that deploys three-dimensional map data, can support access by the end-side device, and is suitable for updating the three-dimensional map.

[0026] The electronic device can receive one or more frames of images captured and uploaded by the above-mentioned end-side device, and store the image as the image to be located. And perform a preliminary analysis on the image to be located to identify whether the image to be located is suitable for updating the three-dimensional map. The preliminary analysis process includes: one or more of position analysis and preliminary analysis of image features on the map to be located. Position analysis includes analyzing whether the location of the image to be located captured by the end-side device belongs to the location covered by the three-dimensional map; preliminary analysis of image features includes analyzing whether the scene contained in the image to be located matches the scene corresponding to the three-dimensional map. When the location of the image to be located is within the location covered by the three-dimensional map, and / or the scene in the image to be located matches the scene of the three-dimensional map, it is determined that the image to be located is suitable for updating the three-dimensional map.

[0027] The electronic device sets a suitable condition for updating the three-dimensional map. When the suitable condition is met, the three-dimensional map is updated. Specifically, the suitable condition may include one or more of the following two: 1) the current time is in the preset three-dimensional map update time period; 2) the resource occupancy rate of the electronic device is higher than the preset resource occupancy rate threshold. The update time period may be any suitable time period set, such as 00:00~8:00 every day. The resource occupancy rate may be the comprehensive occupancy rate of one or more computing resources such as the memory usage rate and the CPU occupancy rate in the electronic device. The resource occupancy rate threshold may be any suitable percentage value set in advance. The electronic device may detect whether the current time is in the three-dimensional map update time period, and / or detect whether the resource occupancy rate of the electronic device is higher than the preset resource occupancy rate threshold. When any one or more of the conditions are met, it is determined that the suitable conditions for updating the three-dimensional map are met.

[0028] The feature extraction model is a pre-set deep learning model or a combination of multiple deep learning models for extracting image feature information, such as ResNet, VGG (Visual Geometry Group), SIFT (Scale-Invariant Feature Transform) algorithm, VLAD (vector of locally aggregated descriptors) algorithm, NetVLAD algorithm, SURF (Speeded Up Robust Features) algorithm, SuperPoint algorithm, LBP (Local Binary Patterns) algorithm, etc. Through the feature extraction model, one or more features such as color information, texture information / edge points, corner points, etc. can be extracted from the image to be located to form image feature information.

[0029] Image feature information represents key information suitable for 3D reconstruction or 3D map update, and specifically, can be key information for matching and / or positioning features stored in a 3D map. Image feature information can be represented in the form of vectors, including one or more of global feature vectors, local feature vectors, etc. Different types of feature information can be extracted using the same or different model algorithms.

[0030] The global feature vector represents the global features of the image, which is used to represent the overall content or style of the image. It is a compact and distinctive representation of the entire image content, with significant visual changes or unique parts. The global feature vector may contain overall information extracted from the image, such as color distribution, texture pattern, and one or more dimensions of information. The local feature vector represents the local features of the image, which is used to reflect the specific details and key points in the image, such as the position, direction, scale, and descriptor (i.e., pixel information or gradient information around the key point) of the key point.

[0031] The global feature vector can be generated by advanced image processing technology (such as using a convolutional neural network CNN combined with a VLAD layer) to perform subsequent matching of the image to be located with the image stored in the cloud map. Different from the global feature vector, the local feature vector focuses on specific areas or key points in the image, which are usually in the image. In the present invention, the local feature vector can be extracted by a detector-based algorithm (such as the SuperPoint algorithm) or a detector-free algorithm (such as the LoFTR algorithm) to achieve more refined feature matching.

[0032] In one embodiment, the feature extraction model includes a global feature extraction model and a local feature extraction model, and the image feature information includes a global feature vector and a local feature vector; calling the feature extraction model to extract image feature information from the image to be located includes: calling the global feature extraction model to extract the global feature vector from the image to be located; calling the local feature extraction model to extract the local feature vector from the image to be located.

[0033] In this embodiment, the local feature extraction model is a model suitable for local feature extraction, which can be a combination of one or more of the SIFT algorithm, SURF algorithm, SuperPoint algorithm, LBP algorithm, etc. For example, the key points and their descriptors in the image to be located can be extracted by algorithms such as SuperPoint to form local feature vectors, and these descriptors contain local information of the key points.

[0034] The global feature extraction model is a model suitable for global feature extraction, which can be a model of deep learning architecture such as ResNet, VGG, VLAD algorithm, NetVLAD algorithm, etc. By using the global feature extraction model, the color distribution, texture pattern, etc. in the image to be located can be captured. By extracting the features of the intermediate layer or output layer of these models, the global feature vector of the image can be obtained.

[0035] For example, the local feature vectors extracted from the image to be located by the SIFT algorithm and / or the SURF algorithm may be aggregated by the VLAD algorithm or the NetVLAD algorithm to form a corresponding global feature vector.

[0036] In this embodiment, by separately setting a model suitable for global feature extraction and a model suitable for local feature extraction, different feature information corresponding to different models are extracted, thereby improving the comprehensiveness, accuracy and efficiency of image feature information extraction.

[0037] Step 120: Calling the image matching model to query the image to be updated that best matches the image feature information from the three-dimensional map to be updated.

[0038] The three-dimensional map to be updated is a pre-built three-dimensional map that needs to be updated. The three-dimensional map contains multiple frames of images, as well as the pose and point cloud data corresponding to each image. The point cloud data includes one or more related data constituting the three-dimensional map, such as 2D feature points and 3D feature points. Specifically, in the process of generating the three-dimensional map, the image feature information of the image is also extracted. The image is the image used in the process of generating the three-dimensional map. The process of extracting the image feature information is similar to the above steps and will not be repeated here.

[0039] The image to be updated is the image in the three-dimensional map that best matches the image to be located. The image to be updated is usually closest and / or has the highest similarity to the pose and / or point cloud data corresponding to the image to be located in the three-dimensional map, and is an image that may need to be replaced. For the extracted image feature information of the image to be located, matching is performed from the three-dimensional map to query the image to be updated. The image matching model can also be any deep learning model suitable for image matching or a combination of multiple deep learning models, which is used to query the image to be updated from the image in the three-dimensional map. The image matching model can have a model that is the same or partially the same as the feature extraction model. For example, the image matching model can be one or more of the SIFT algorithm, SURF algorithm, ORB (Oriented FAST and Rotated BRIEF) algorithm, NetVLAD algorithm, SuperPoint algorithm, etc.

[0040] The image feature information of the image to be located is matched with the image information of the image in the three-dimensional map through the image matching model, and multiple frames of images with the highest matching degree are queried as candidate images. Feature matching is performed between the image to be located and each frame of the candidate image. Based on the matching results, the image to be updated is determined from the candidate images to obtain a set of feature point matching pairs between the image to be located and the image to be updated.

[0041] Specifically, a feature point matching pair set between the image to be located and each candidate image frame can be calculated, and the image to be updated can be determined from the candidate image based on the feature point matching pair set. The feature point matching pair set represents a set of two-dimensional feature point pairs that have a matching relationship between two frames of images, and the two-dimensional feature point pairs represent the same or similar physical point pairs in the two images.

[0042] Step 130: Identify the first pose of the image to be located in the three-dimensional map to be updated based on the image to be updated.

[0043] The first pose represents the pose of the image to be updated in the three-dimensional map. The electronic device stores the pose corresponding to each frame of the image (including the image to be updated) in the process of pre-building the three-dimensional map. Based on the pose of the image to be updated and the feature point matching pair set between the image to be updated and the image to be located, the pose of the image to be located can be calculated.

[0044] Specifically, the transformation relationship between the image to be located and the image to be updated can be calculated according to a preset relevant geometric transformation method (such as RANSAC combined with a perspective transformation matrix), and then the first posture of the image to be located can be derived according to this transformation relationship, a set of feature point matching pairs, and the posture of the image to be updated.

[0045] Step 140: Calculate point cloud data of the image to be positioned based on the first pose.

[0046] In this embodiment, the electronic device can calculate the point cloud data of the image to be located in the three-dimensional map based on the calculated first pose and the image to be located. The pixel coordinates of the image to be located on the two-dimensional image can be converted into point cloud coordinates in the three-dimensional space by using the camera intrinsic parameters (such as focal length, principal point coordinates, etc.) and extrinsic parameters (such as rotation and translation matrices, etc.) in the terminal device through projective geometry calculation, thereby calculating the point cloud data.

[0047] Specifically, the three-dimensional point set can be restored through triangulation calculation to obtain the 3D feature points of the image to be located.

[0048] Step 150, updating the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated.

[0049] After calculating the first pose and point cloud data of the image to be located, the point cloud data can be updated to the point cloud set in the three-dimensional map. At the same time, the first pose, image feature information and the image to be located are saved to achieve a partial update of the three-dimensional map.

[0050] The specific update process may include inserting new image nodes, updating the posture parameters of image nodes, and merging new point cloud data, so that the three-dimensional map remains up to date and accurately reflects changes in the actual environment. It can add previously unrecorded area information to the three-dimensional map and update map information in areas where environmental changes have occurred.

[0051] The updating method of the three-dimensional map constructed based on the panoramic image in the present application extracts image feature information of the image to be positioned, combines the image feature information of the image in the existing three-dimensional map, queries the image to be updated in the three-dimensional map that best matches the image to be positioned, and calculates the first pose of the image to be positioned in combination with the queried image to be updated. This can improve the accuracy of the first pose calculation and realize high-precision real-time positioning of the image to be positioned, avoiding the problem of inaccurate positioning when the position of the image to be positioned is too far from the three-dimensional reconstruction scanning trajectory.

[0052] Furthermore, under the premise of ensuring the accuracy of the first pose calculation, the point cloud data of the image to be located is calculated based on the first pose, and then the three-dimensional image is updated based on the point cloud data, thereby continuously optimizing and improving the accuracy and real-time performance of the three-dimensional map, and improving the real-time performance and update quality of the three-dimensional image update. In addition, the present application can realize automatic three-dimensional map updates without the need for manual user operation updates, thereby improving the flexibility and update efficiency of map updates.

[0053] In one embodiment, the image matching model includes an image retrieval model and a feature matching model. Figure 2 As shown, step 120 includes: Step 210: Calling an image retrieval model to query a first number of candidate images that match the image feature information from the three-dimensional map to be updated.

[0054] Step 220: Call the image matching model to calculate a set of feature point matching pairs between the image to be located and each candidate image.

[0055] Step 230 : determining an image to be updated from the candidate images based on the feature point matching pair set.

[0056] In this embodiment, the image retrieval model is used to calculate candidate images, and the feature matching model is used to perform feature matching. The image retrieval model, the feature matching model, and the feature extraction model can be completely identical or partially identical models. For example, the image retrieval model and the feature extraction model both use models based on the NetVLAD algorithm, and the feature matching model and the feature extraction model both use models based on the SuperPoint algorithm, the LoFTR (Local Feature Transformer) algorithm, and one or more thereof. The same algorithms used in the image retrieval model, the feature matching model, and the feature extraction model can have the same parameters or different parameters. When different parameters are used, targeted optimization can be performed for the required purpose of feature extraction, feature matching, image matching, etc., thereby improving the accuracy and efficiency of feature extraction, feature matching, and image matching.

[0057] Specifically, the image feature information includes global feature vectors and local feature vectors. The global feature vectors and local feature vectors of each image in the three-dimensional map can be pre-extracted through the above process, and the model formed based on the NetVLAD algorithm is called to analyze the global feature vectors and local feature vectors of the image to be located and the global feature vectors and local feature vectors of each image in the three-dimensional map, and identify a first number of images from each image in the three-dimensional map as candidate images. The candidate image is a multi-frame image with the highest matching degree with the image to be located. The first number can be any suitable fixed number set in advance, or an adaptive number set based on the matching degree. For example, the first number can be fixedly set to any suitable number such as 5, 8, 10, 15, 20, etc., or the number of images whose matching degree reaches a preset matching degree threshold can be directly used as the first number. The matching degree reflects the similarity between the candidate image and the image to be located.

[0058] After the candidate images are obtained, the feature point matching pair sets between each candidate image and the image to be located are calculated respectively. Specifically, the feature points of the two frames of images (i.e., the image to be located and one of the candidate images) are extracted through the image matching model, the similarity between the descriptors (such as Euclidean distance, cosine similarity, etc.) is calculated, and the matching feature point pairs are found to form a feature point matching pair set of the two frames of images. In addition, matching can also be performed by calculating statistical features such as the grayscale histogram or grayscale co-occurrence matrix of the image to obtain a feature point matching pair set of the two frames of images. Alternatively, the two frames of images can be converted from the spatial domain to the transform domain (such as the Fourier transform domain, the wavelet transform domain, etc.), and then matched in the transform domain to obtain a feature point matching pair set of the two frames of images.

[0059] The corresponding global feature point matching pair subset is obtained through the global feature vector, and the corresponding local feature point matching pair subset is obtained through the local feature vector. After obtaining the feature point matching pair set between each candidate image and the image to be located, the image matching model is further called to determine the image to be updated based on the calculated feature point matching pair set.

[0060] In this embodiment, a first number of candidate images are first screened out, and then a set of feature point matching pairs between each candidate image and the image to be located is calculated. Based on the set of feature point matching pairs, the image to be updated is screened out from the candidate images, thereby improving the accuracy of screening the image to be updated.

[0061] In one embodiment, step 220 includes: identifying the scene of the image to be located, the scene including indoor scenes and outdoor scenes; calling an image matching model matching the scene to calculate a set of feature point matching pairs between the image to be located and each candidate image.

[0062] In this embodiment, scene recognition can be performed on the image to be located. By using the image feature information such as color, texture, spatial layout, lighting pattern, etc. in the early analysis, the scene objects contained in the image can be further identified. The scene objects may include furniture, lamps, decorations, buildings, natural landscapes, roads, etc., and it can be determined whether the image to be located is in an outdoor scene or an indoor scene.

[0063] Different image matching models are set for different scenes. The image matching models include models based on detector algorithms and models based on non-detector algorithms. When the image to be located is in an indoor scene, the model based on the detector algorithm is used to calculate the feature point matching pair set; when the image to be located is in an outdoor scene, the model based on the non-detector algorithm is used to calculate the feature point matching pair set.

[0064] Specifically, the detection algorithm may be the SuperPoint algorithm, and the detector-free algorithm may be the LoFTR algorithm. By calling different image matching models based on different scenes to calculate the feature point matching pair set, the accuracy and efficiency of the feature point matching pair set calculation can be improved.

[0065] In one embodiment, step 220 includes: calling a model based on a detector algorithm to calculate an initial set of feature point matching pairs between the image to be located and each candidate image, detecting whether the initial set of feature point matching pairs meets the expected feature distribution, and when it meets the feature distribution, using the initial set of feature point matching pairs as the feature point matching pair set; when it does not meet the feature distribution, calling a model based on a detector-free algorithm to calculate a set of feature point matching pairs between the image to be located and each candidate image.

[0066] In this embodiment, the model based on the algorithm with a detector is preferentially used to calculate the feature point matching pair set, and the feature point matching pair set output by the model based on the algorithm with a detector is used as the initial set of feature point matching pairs. It is detected whether the initial set meets the expected feature distribution. If it meets the requirements, the initial combination obtained based on the algorithm with a detector is used as the final feature point matching pair set; if it does not meet the requirements, the algorithm based on the algorithm without a detector is used, and the feature point matching pair set obtained based on the algorithm without a detector is used as the final feature point matching pair set.

[0067] The feature distribution includes one or more of the number of feature point matching pairs, the central tendency (such as mean, median, etc.) characteristics presented by the set of feature point matching pairs, the dispersion (such as variance, standard deviation) characteristics, and the shape (such as skewness, kurtosis) characteristics.

[0068] Specifically, corresponding to the image feature information, the feature point matching pairs may also include one or both of the local feature point matching pair subsets and the global feature point matching pair subsets. When the initial set of feature point matching pairs is calculated using a detector-based algorithm model, the local feature vector of the image to be located and the local feature vector of the candidate image are used as inputs of the model, and by running the model, a local feature point matching pair subset corresponding to the image to be located and the candidate image is output, and the subset is used as the above-mentioned initial set. When the initial set of feature point matching pairs is calculated using a detector-free algorithm model, the local feature vector and global feature vector of the image to be located and the local feature vector and global feature vector of the candidate image are used as inputs of the model, and by running the model, a local feature point matching pair subset and a global feature point matching pair subset corresponding to the image to be located and the candidate image are output, and the two subsets are used as the feature point matching pair set.

[0069] In one embodiment, step 130 includes: based on the feature point matching pair set between the image to be positioned and the image to be updated, calling a preset random sample consensus algorithm to calculate an inlier matching pair set; and calculating the first pose based on the inlier matching pair set.

[0070] In this embodiment, the inliers refer to reliable feature points. Compared with the external points containing noise, the accuracy of the pose calculation can be improved by performing pose calculation through the internal points. The electronic device pre-sets the random sample consensus (RANSAC) algorithm, takes the feature point matching pair set as the input of the algorithm, and runs the algorithm to output the corresponding inliers matching pair set. Specifically, the RANSAC algorithm is used to verify the matching results of the feature points in the image to be positioned and the image to be updated from the feature point matching pair set, and the external points belonging to the noise are excluded. The retained feature point matching pairs constitute the inliers matching pair set.

[0071] After obtaining the set of inlier matching pairs, the set of inlier matching pairs is used as the input of the relevant 3D reconstruction algorithm model to calculate the first pose of the image to be located. The 3D reconstruction algorithm model can be a triangulation model or a PnP algorithm model, through which an accurate first pose is obtained. The first pose represents the rotation and translation relationship of the image to be located relative to the 3D map.

[0072] In one embodiment, before step 150, the method further includes: determining whether the image to be located satisfies a first update condition based on the first number of point pairs in a set of feature point matching pairs and / or the second number of point pairs in a set of inner point matching pairs between the image to be located and the image to be updated; when the first update condition is met, updating the image to be located, the first pose and the point cloud data to the three-dimensional map to be updated.

[0073] In this embodiment, the electronic device further sets a first update condition, which may include any one or more of the following two combinations: 1) the number of first point pairs exceeds the first number threshold; 2) the number of second point pairs exceeds the second number threshold. The first number threshold and the second number threshold may be any appropriate numerical values, and the numerical values ​​of the two may be the same or different. For example, the first number threshold is greater than the second number threshold. When any one of the above two conditions is met, or only when both conditions are met, it is determined that the first update condition is met; otherwise, it is determined that the first update condition is not met.

[0074] For the feature point matching pair sets of the image to be located and the image to be updated, the number of feature point matching pairs therein (i.e., the first point pair number) can be counted to detect whether the number exceeds the first number threshold, and the number of inlier point matching pairs in the inlier point matching pair sets of the two (i.e., the second point pair number) can be counted to detect whether the number exceeds the second number threshold. Optionally, when both are judged as yes, it is determined that the first update condition is satisfied; when any one of them is judged as no, it is determined that the first update condition is not satisfied.

[0075] When the first update condition is met, it means that the image to be located is an image related to the scene in the three-dimensional map to be updated, or is related to the field of view of the scene in the three-dimensional map, and step 150 is executed. Otherwise, it means that the image to be located is not related to the scene in the three-dimensional map to be updated, or the correlation is not high, and it does not need to be updated to the three-dimensional map.

[0076] By setting the first update condition, it can be identified whether the image to be located is related to the field of view of the three-dimensional map, thereby avoiding updating images irrelevant to the scene corresponding to the three-dimensional map into the three-dimensional map, thereby reducing the quality of the three-dimensional map.

[0077] In one embodiment, before step 150, the method further includes: calculating the difference between the first pose and the existing pose in the three-dimensional map to be updated; determining whether the image to be located meets the second update condition based on the difference; and when the second update condition is met, updating the image to be located, the first pose and the point cloud data to the three-dimensional map to be updated.

[0078] In this embodiment, the difference between the calculated first pose and the pose of one or more existing maps in the three-dimensional map (i.e., the existing pose) can be compared. The higher the difference, the lower the probability that there is an image or point cloud information similar to the first pose in the three-dimensional map. Conversely, the lower the difference, the greater the probability that there is an image or point cloud information similar to the first pose, that is, in the three-dimensional map, there is no corresponding image and point cloud data within a certain position range of the first pose.

[0079] The second update condition may be: the corresponding difference exceeds a preset first difference threshold. The first difference threshold may be any pre-set appropriate value.

[0080] Specifically, the difference between the first pose and the existing poses of each candidate image may be calculated, and the smallest difference may be detected to determine whether it satisfies the second update condition. If so, step 150 is performed. When the smallest difference does not satisfy the second update condition, step 150 is not performed.

[0081] In one embodiment, the difference between the first pose and the pose of the image to be updated may be calculated, and when the difference is greater than a preset first difference threshold, it is determined that the second update condition is met.

[0082] In one embodiment, step 150 includes: when the second update condition is met, the image to be located, the first pose and the point cloud data are added to the three-dimensional map to be updated; when the second update condition is not met, and the pose difference between the image to be updated and the image to be located is less than a preset second difference threshold, the information of the image to be updated in the three-dimensional map to be updated is replaced with the image to be located, the first pose and the point cloud data to realize the update of the three-dimensional map to be updated, and the second difference threshold is less than the first difference threshold.

[0083] In this embodiment, updating the three-dimensional map includes adding map data related to the image to be located (including one or more of the image to be located, the first pose, point cloud data, image feature information, etc.) to the three-dimensional map, and replacing the map data in the three-dimensional map. Specifically, it includes adding the image to be located and its first pose and point cloud data to the three-dimensional map without changing other images in the three-dimensional map and their corresponding poses and point cloud data; it may also include replacing the corresponding image in the three-dimensional map and its pose and point cloud data using the image to be located and its first pose and point cloud data.

[0084] In this embodiment, the second difference threshold is a preset value smaller than the first difference threshold. When the difference between the postures corresponding to the two frames of images is smaller than the second difference threshold, it means that the postures of the two frames are close enough, and the two frames of images can be regarded as images at the same position.

[0085] When the first update condition and the second update condition are met, it means that there is no corresponding map data at a posture near the first posture. The map data corresponding to the image to be located can be added to the three-dimensional map to be updated, so that the data in the three-dimensional map is more complete.

[0086] When the first update condition is met, but the second update condition is not met, it means that there is corresponding map data within a certain range near the first pose. At this time, it can be further detected whether the pose difference between the image to be located and the image to be updated is less than the second difference threshold. If so, it means that the map data related to the image to be updated in the map data is not the latest map data. Since the image to be located is the latest image, the map data corresponding to the image to be updated can be replaced with the map data of the image to be located, thereby realizing the update of the three-dimensional map to be updated.

[0087] In one embodiment, Figure 3 As shown, another method for updating a three-dimensional map constructed based on a panoramic image is provided, the method comprising: Step 301: obtain an image to be positioned, and call a feature extraction model to extract image feature information from the image to be positioned.

[0088] In this embodiment, the image to be located may be an image of a scene in the three-dimensional map to be updated, which is taken by the terminal device. The scene of the three-dimensional map may include a combination of one or more indoor scenes and outdoor scenes.

[0089] Image feature information may include global feature vectors and local feature vectors. Specifically, a pre-trained convolutional neural network model is called to extract local feature vectors from the image to be located to ensure the accuracy of subsequent image matching; the VLAD layer in the ResNet model or VGG model is called to extract global feature vectors from the image to be located.

[0090] Step 302: Calling an image retrieval model to query a first number of candidate images matching the image feature information from the three-dimensional map to be updated.

[0091] Optionally, the image retrieval model may be a model formed based on the NetVLAD algorithm, and for the global feature vector and local feature vector of the image to be located, the model is called to query a first number of candidate images that best match (eg, are most similar to) the image to be located from the three-dimensional map to be updated.

[0092] Step 303: Call the image matching model to calculate a feature point matching pair set between the image to be located and each candidate image, and determine the image to be updated from the candidate images based on the feature point matching pair set.

[0093] The image to be located and each candidate image form an image frame pair, thereby forming a first number of image frame pairs. Identify whether the scene to be located belongs to an outdoor scene or an indoor scene. When it belongs to an outdoor scene, call the image matching model formed based on the SuperPoint algorithm to calculate the feature point matching pair set of each image frame pair. When it belongs to an indoor scene, call the image matching model formed based on the LoFTR algorithm to calculate the feature point matching pair set of each image frame pair.

[0094] After obtaining the first number of feature point matching pair sets, determine from these feature point matching pair sets which frame of candidate images is suitable as the image to be updated.

[0095] Step 304 : Based on the feature point matching pair set between the image to be positioned and the image to be updated, a preset random sample consensus algorithm is called to calculate an inlier matching pair set.

[0096] Specifically, the PnP algorithm can be combined with a random sampling consensus algorithm to verify the feature point matching pair set between the image to be positioned and the image to be updated, and the inlier matching pairs can be extracted from the feature point matching pairs to form an inlier matching pair set.

[0097] Step 305: Calculate the first pose based on the set of inlier matching pairs.

[0098] Step 306: Calculate point cloud data of the image to be positioned based on the first pose.

[0099] Step 307 : determining whether the image to be located satisfies a first update condition based on the first number of point pairs in the feature point matching pair set and / or the second number of point pairs in the interior point matching pair set between the image to be located and the image to be updated.

[0100] In this embodiment, when the first update condition is met, the following step 308 is executed. When the first update condition is not met, the updating of the three-dimensional image with the positioning image is terminated, or the image to be positioned is discarded, and other images are continuously identified to determine whether they are suitable for updating the three-dimensional image.

[0101] Step 308: Calculate the difference between the first pose and the existing poses in the three-dimensional map to be updated.

[0102] Step 309, determining whether the image to be located meets the second update condition based on the difference; When the second update condition is met, step 310 is executed. Otherwise, the updating of the three-dimensional image by the positioning image is terminated, or the image to be positioned is discarded, and other images are continuously identified to determine whether they are suitable for updating the three-dimensional image. Or when the second update condition is not met, step 310 is still executed, but the updating method for the three-dimensional map is different.

[0103] In one embodiment, the execution order between the above-mentioned step 306 and steps 307~309 can be swapped. For example, step 307 can be executed first, and when the first update condition is met, steps 306 and 308~309 are executed, or when the first update condition is met, steps 308~309 are executed, and when the second update condition is further met, step 306 is executed. In this way, when it is identified that the image to be positioned does not meet the update condition, there is no need to perform point cloud data calculation.

[0104] Step 310: update the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated.

[0105] In one embodiment, when the second update condition is met, the image to be located, the first pose and the point cloud data are added to the three-dimensional map to be updated; When the second update condition is not met, further analyze whether the posture difference between the image to be updated and the image to be positioned is less than the second difference threshold. If so, the information of the image to be updated in the three-dimensional map to be updated is replaced with the image to be positioned, the first posture and the point cloud data to realize the update of the three-dimensional map to be updated. The second difference threshold is less than the first difference threshold.

[0106] The updating method of the three-dimensional map constructed based on the panorama in the present application calculates the pose of the image to be located by combining the set of inner point matching pairs between the image to be located and the image to be updated, and combines the pose calculation of the image to be located with the data of the existing three-dimensional map, which can improve the accuracy of the pose calculation of the image to be located. In addition, by setting corresponding update conditions to update the map, the quality and timeliness of the map update can be improved.

[0107] In one embodiment, Figure 4 As shown, a device for updating a three-dimensional map constructed based on a panoramic image is provided, the device comprising: The feature extraction module 410 is used to obtain the image to be located and call the feature extraction model to extract image feature information from the image to be located.

[0108] The image matching module 420 is used to call the image matching model to query the image to be updated that best matches the image feature information from the three-dimensional map to be updated.

[0109] The pose calculation module 430 is used to identify the first pose of the image to be located in the three-dimensional map to be updated based on the image to be updated.

[0110] The map updating module 440 is used to calculate the point cloud data of the image to be located based on the first pose, and update the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated.

[0111] In one embodiment, the image matching module 420 is also used to call the image retrieval model to query a first number of candidate images that match the image feature information from the three-dimensional map to be updated; call the image matching model to calculate a set of feature point matching pairs between the image to be located and each candidate image; and determine the image to be updated from the candidate images based on the set of feature point matching pairs.

[0112] In one embodiment, the image matching module 420 is also used to identify the scene of the image to be located, including indoor scenes and outdoor scenes; and call the image matching model that matches the scene to calculate the set of feature point matching pairs between the image to be located and each candidate image.

[0113] In one embodiment, the pose calculation module 430 is also used to call a preset random sample consensus algorithm to calculate an inlier matching pair set based on a feature point matching pair set between the image to be positioned and the image to be updated; and calculate the first pose based on the inlier matching pair set.

[0114] In one embodiment, Figure 5 As shown, the above device also includes: An update condition judgment module 450 is used to determine whether the image to be located satisfies a first update condition based on the number of first point pairs in the feature point matching pair set and / or the number of second point pairs in the interior point matching pair set between the image to be located and the image to be updated; The map updating module 440 is also used to update the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated when the first update condition is met.

[0115] In one embodiment, the update condition judgment module 450 is further used to calculate the difference between the first pose and the existing pose in the three-dimensional map to be updated; and determine whether the image to be located meets the second update condition based on the difference.

[0116] The map updating module 440 is also used to update the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated when the second update condition is met.

[0117] In one embodiment, the second update condition includes that the difference exceeds a preset first difference threshold.

[0118] The map update module 440 is also used to add the image to be located, the first pose and the point cloud data to the three-dimensional map to be updated when the second update condition is met; when the second update condition is not met and the pose difference between the three-dimensional map to be updated and the most matching map is less than a preset second difference threshold, the information of the image to be updated in the three-dimensional map to be updated is replaced with the image to be located, the first pose and the point cloud data to realize the update of the three-dimensional map to be updated, and the second difference threshold is less than the first difference threshold.

[0119] In one embodiment, the feature extraction model includes a global feature extraction model and a local feature extraction model, and the image feature information includes a global feature vector and a local feature vector. The feature extraction module 410 is also used to call the global feature extraction model to extract a global feature vector from the image to be located; and call the local feature extraction model to extract a local feature vector from the image to be located.

[0120] In one embodiment, a computer storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments.

[0121] In one embodiment, an electronic device is also provided, including one or more processors; a memory, wherein one or more programs are stored in the memory, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the steps in the above-mentioned method embodiments. For example, the electronic device may be a cloud server that deploys three-dimensional map data.

[0122] In one embodiment, Figure 6As shown, it shows a schematic diagram of the structure of an electronic device for implementing an embodiment of the present application. The electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0123] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0124] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, including a computer-readable medium carrying instructions, in such an embodiment, the instructions can be downloaded and installed from a network through a communication part 609, and / or installed from a removable medium 611. When the instructions are executed by a central processing unit (CPU) 601, the various method steps described in the present application are executed.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0126] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, all of the above embodiments may be used in any combination. The information disclosed in this background technology section is intended only to deepen the understanding of the overall background technology of the present application and should not be regarded as an admission or in any form to imply that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for updating a three-dimensional map constructed based on a panoramic image, characterized in that: The method comprises: Acquire an image to be located, and call a feature extraction model to extract image feature information from the image to be located; Calling the image matching model to query the image to be updated that best matches the image feature information from the three-dimensional map to be updated; Identify the first pose of the image to be located in the three-dimensional map to be updated based on the image to be updated; Calculating point cloud data of the image to be positioned based on the first posture; The image to be located, the first pose and the point cloud data are updated into the three-dimensional map to be updated.

2. The updating method according to claim 1, characterized in that: The image matching model includes an image retrieval model and a feature matching model. The calling of the image matching model to query the image to be updated that best matches the image feature information from the three-dimensional map to be updated includes: Calling the image retrieval model to query a first number of candidate images matching the image feature information from the three-dimensional map to be updated; Calling the image matching model to calculate a set of feature point matching pairs between the image to be located and each candidate image; An image to be updated is determined from the candidate images based on the feature point matching pair set.

3. The updating method according to claim 2, characterized in that: The calling the image matching model to calculate a set of feature point matching pairs between the image to be located and each candidate image includes: Identifying a scene of the image to be located, wherein the scene includes an indoor scene and an outdoor scene; An image matching model matching the scene is called to calculate a set of feature point matching pairs between the image to be located and each candidate image.

4. The updating method according to claim 2, characterized in that: The step of identifying the first pose of the image to be located in the three-dimensional map to be updated based on the image to be updated includes: Based on the feature point matching pair set between the image to be positioned and the image to be updated, a preset random sample consensus algorithm is called to calculate an interior point matching pair set; The first pose is calculated based on the set of interior point matching pairs.

5. The updating method according to claim 4, characterized in that: Before updating the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated, the method further includes: Determining whether the image to be located satisfies a first update condition based on a first number of point pairs in a feature point matching pair set between the image to be located and the image to be updated and / or a second number of point pairs in the interior point matching pair set; When the first update condition is met, the step of updating the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated is performed.

6. The updating method according to any one of claims 1 to 5, characterized in that: Before updating the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated, the method further includes: Calculating the difference between the first posture and the existing posture in the three-dimensional map to be updated; Determining whether the image to be located satisfies a second update condition based on the difference; When the second update condition is met, the image to be located, the first pose and the point cloud data are updated into the three-dimensional map to be updated.

7. The updating method according to claim 6, characterized in that: The second update condition includes that the difference exceeds a preset first difference threshold; The updating of the image to be located, the first pose and the point cloud data into the three-dimensional map to be updated comprises: When the second update condition is met, adding the image to be located, the first pose and the point cloud data to the three-dimensional map to be updated; When the second update condition is not met and the posture difference between the image to be updated and the image to be positioned is less than a preset second difference threshold, the information of the image to be updated in the three-dimensional map to be updated is replaced by the image to be positioned, the first posture and the point cloud data to achieve the update of the three-dimensional map to be updated, and the second difference threshold is less than the first difference threshold.

8. The updating method according to claim 1, characterized in that: The feature extraction model includes a global feature extraction model and a local feature extraction model, and the image feature information includes a global feature vector and a local feature vector; The calling of the feature extraction model to extract image feature information from the image to be located includes: Calling a global feature extraction model to extract a global feature vector from the image to be located; A local feature extraction model is called to extract a local feature vector from the image to be located.

9. A device for updating a three-dimensional map constructed based on a panoramic image, characterized in that: The device comprises: A feature extraction module is used to obtain an image to be located and call a feature extraction model to extract image feature information from the image to be located; An image matching module, used for invoking an image matching model to query an image to be updated that best matches the image feature information from the three-dimensional map to be updated; A posture calculation module, used for identifying the first posture of the image to be located in the three-dimensional map to be updated based on the image to be updated; A map updating module is used to calculate the point cloud data of the image to be located based on the first posture, and update the image to be located, the first posture and the point cloud data into the three-dimensional map to be updated.

10. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to execute the method for updating a three-dimensional map constructed based on a panoramic image as described in any one of claims 1 to 8.

Images