Method and apparatus for searching 3D map

Abstract

A 3D map retrieval method and device are beneficial to improving retrieval performance and belong to the technical field of 3D maps. The 3D map retrieval method comprises the following steps: acquiring binary data of S 3D map descriptors, performing i-level retrieval in the binary data of the S 3D map descriptors according to the binary data of the retrieval descriptor (S101A), screening to obtain P 3D map descriptors (S102A), performing m-level decompression on the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors (S103A), and performing j-level retrieval in the reconstructed data of the P 3D map descriptors according to part or all of the data of the retrieval descriptor to obtain Q 3D map descriptors (S104A). The retrieval speed can be improved, and the retrieval accuracy can be guaranteed.

Description

Technical Field

[0001] This application relates to positioning technology, and more particularly to a method and apparatus for retrieving 3D maps. Background Technology

[0002] Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) are emerging multimedia virtual scene technologies in recent years. These technologies can create virtual reality and overlay it onto the real world to produce new visual environments and interactive experiences. In these applications, electronic devices need to determine their own pose information in the current environment to accurately blend virtual objects with the real scene.

[0003] On the other hand, in applications such as autonomous driving, autonomous navigation, drone automatic inspection, and industrial robots, vehicles, drones, robots and other vehicles need to determine their own position in the current environment by determining the position of the electronic equipment they carry in order to perform precise path planning, navigation, detection and control.

[0004] For the above applications, where it is necessary to determine the pose of an electronic device in the current environment, a typical solution is as follows: the electronic device receives a 3D map of its environment from a server or other device, then collects visual information about the environment through local sensors, and searches the downloaded 3D map based on the collected visual information to determine the current pose of the electronic device.

[0005] However, original 3D maps typically contain a huge amount of data, and the computational load for retrieval is usually enormous, requiring a lot of computing resources and taking a long time, which affects the user experience. Summary of the Invention

[0006] This application provides a 3D map retrieval method and apparatus that can improve retrieval performance.

[0007] In a first aspect, an embodiment of the present application provides a method for retrieving a 3D map. The method may include: extracting the binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in the 3D map. Performing an i-level retrieval on the binary data of the S 3D map descriptors according to the binary data of the retrieval descriptor to obtain P 3D map descriptors. The retrieval descriptor is a feature corresponding to the real environment extracted from the visual information collected by the sensor of the electronic device. Performing an m-level decompression on the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors. The decompression process (complete decompression process) of the compressed data of the P 3D map descriptors at least includes the m-level decompression. Performing a j-level retrieval on the reconstructed data of the P 3D map descriptors according to partial data or all data of the retrieval descriptor to obtain Q 3D map descriptors. S, P, Q, i, j, and m are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, where T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the total number of levels or the threshold of the levels of the 3D map retrieval. L is a positive integer greater than 1.

[0008] It can be seen that in this embodiment, according to the binary data of the retrieval descriptor, an i-level retrieval is performed on the binary data of the S 3D map descriptors to screen out P 3D map descriptors. An m-level decompression is performed on the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors. According to partial data or all data of the retrieval descriptor, a j-level retrieval is performed on the reconstructed data of the P 3D map descriptors to screen out a smaller number of 3D map descriptors. One level of retrieval in at least two levels of retrieval uses the binary data in the compressed data, which can improve the retrieval speed, and the other level of retrieval uses the reconstructed data obtained by decompression, which can ensure the retrieval accuracy. Compared with retrieving in the reconstructed data of the 3D map obtained by complete decompression, the 3D map retrieval method provided by the embodiment of the present application can improve the retrieval performance. In other words, the 3D map retrieval method provided by the embodiment of the present application can improve the retrieval performance.

[0009] The j-level can be an intermediate level in multiple levels of retrieval, that is, j < L, or it can also be the last level in multiple levels of retrieval, that is, j = L.

[0010] In one example, the above m-level decompression can be a staged decompression, which can be understood as a sub-process of complete decompression or as partial decompression. In another example, the above m-level decompression can be complete decompression.

[0011] In one possible design, N out of Q 3D map descriptors are used for localization, and the 3D map points corresponding to the N 3D map descriptors are matched with the 3D map points corresponding to the search descriptors, where N is a positive integer, 0 < N ≤ Q. The above method may also include: performing localization based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0012] In this embodiment, when the 3D map points corresponding to the N retrieved 3D map descriptors match the map points corresponding to the retrieved descriptors, positioning can be performed based on the 3D map points corresponding to the N 3D map descriptors, which helps to improve positioning performance.

[0013] In one possible design, the retrieval method used for the i-th level is based on the first distance, and the retrieval method used for the j-th level is based on the second distance.

[0014] The first distance can include distances obtained using binarized data, such as Hamming distance. For example, the first distance can be Hamming distance. Alternatively, the first distance can be the sum of the absolute differences between corresponding bits of two binarized data sets. Retrieval based on the first distance refers to determining relevance or similarity by calculating the first distance to filter 3D map descriptors. The second distance can include, but is not limited to, Euclidean distance, inner product distance, cosine distance, Manhattan distance, etc. Retrieval based on the second distance refers to determining relevance or similarity by calculating the second distance to filter 3D map descriptors.

[0015] In this embodiment, searching based on the first distance can improve the search speed, while searching based on the second distance can improve the search accuracy.

[0016] In one possible design, the method may further include: receiving a retrieval descriptor and binarizing the retrieval descriptor to obtain binarized data of the retrieval descriptor. The execution entity (which may be an electronic device or a server (e.g., a cloud server)) performing the retrieval method described in the first aspect can receive retrieval descriptors sent by other devices. For example, the execution entity performing the retrieval method described in the first aspect receives retrieval descriptors collected and extracted by other electronic devices. The retrieval descriptor is then binarized to obtain binarized data of the retrieval descriptor.

[0017] In a possible design, the method may further include: receiving the visual information, extracting a retrieval descriptor from the visual information, and performing binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor; an execution entity (which may be an electronic device or a server) for executing the retrieval method described in the first aspect may receive the visual information sent by other devices and extract a retrieval descriptor therefrom. Then, binarization processing is performed on the retrieval descriptor to obtain binarized data of the retrieval descriptor.

[0018] In a possible design, the method may further include: in response to an operation of collecting visual information input by a user, triggering the sensor to collect visual information of the real environment, obtaining the visual information, extracting the retrieval descriptor from the visual information, and performing binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor. An execution entity (which may be an electronic device or a server) for executing the retrieval method described in the first aspect may collect the visual information and extract a retrieval descriptor therefrom. Then, binarization processing is performed on the retrieval descriptor to obtain binarized data of the retrieval descriptor.

[0019] The 3D map retrieval method of the embodiments of the present application can retrieve retrieval descriptors in different application scenarios in a 3D map, and has wide applicability.

[0020] In a possible design, when N < Q, the method further includes: decompressing the compressed data of Q 3D map descriptors at the k-th level to obtain reconstructed data of Q 3D map descriptors. According to partial data or all data of the retrieval descriptor, performing an r-th level retrieval in the reconstructed data of Q 3D map descriptors to obtain N 3D map descriptors. Wherein, N < Q < P, the decompression process of the compressed data of P 3D map descriptors includes an m-th level decompression (phased decompression) and a k-th level decompression (phased decompression), r and k are positive integers, m < k, j < r ≤ L.

[0021] In this embodiment, based on the binarized data of the search descriptors, a level i search is performed on the binarized data of S 3D map descriptors to obtain P 3D map descriptors. The compressed data of the P 3D map descriptors is then decompressed at level m (staged decompression) to obtain reconstructed data for the P 3D map descriptors. Based on partial or complete data from the search descriptors, a level j search is performed on the reconstructed data of the P 3D map descriptors to obtain Q 3D map descriptors. Next, the compressed data of the Q 3D map descriptors is decompressed at level k (staged decompression) to obtain reconstructed data for the Q 3D map descriptors. Based on all data from the search descriptors, a level r search is performed on the reconstructed data of the Q 3D map descriptors to obtain N 3D map descriptors, which can be used for positioning. At least one of the three levels of retrieval uses the binarized data from the compressed data, which can improve retrieval speed. The other two levels of retrieval use reconstructed data obtained from different decompressions, which can improve retrieval speed while ensuring retrieval accuracy.

[0022] Specifically, the reconstructed data obtained from the m-th level decompression and the k-th level decompression have different degrees of decompression or different degrees of distortion.

[0023] In one possible design, the compressed data of P 3D map descriptors includes the binarized and quantized data of each of the P 3D map descriptors. The m-th level decompression of the compressed data of the P 3D map descriptors yields the reconstructed data of the P 3D map descriptors, including: inverse quantization of the quantized data of each of the P 3D map descriptors to obtain P inverse quantized data, which are used as the reconstructed data of the P 3D map descriptors. The k-th level decompression of the compressed data of Q 3D map descriptors yields the reconstructed data of Q 3D map descriptors, including: inverse quantization of the quantized data of each of the Q 3D map descriptors to obtain Q inverse quantized data; and the reconstructed data of each of the Q 3D map descriptors is obtained based on the Q inverse quantized data and the binarized data of each of the Q 3D map descriptors.

[0024] In this embodiment, by inverse quantization of the quantized data of each of the P 3D map descriptors, inverse quantized data of each of the P 3D map descriptors is obtained. Based on the inverse quantized data of each of the P 3D map descriptors (used as reconstructed data for this level of retrieval), a level j retrieval is performed, resulting in Q 3D map descriptors. By inverse quantization of the quantized data of each of the Q 3D map descriptors, Q inverse quantized data are obtained. Based on the Q inverse quantized data and the binarized data of each of the Q 3D map descriptors, reconstructed data of each of the Q 3D map descriptors is obtained. Based on the reconstructed data of each of the Q 3D map descriptors, a level r retrieval is performed to obtain N 3D map descriptors. The distortion of the reconstructed data from the level j retrieval is greater than that from the level r retrieval; that is, a coarse retrieval is performed first to improve retrieval speed, followed by a fine retrieval to improve retrieval accuracy.

[0025] In one possible design, when N=Q, the compressed data of P 3D map descriptors includes the binarized data and quantized data of each of the P 3D map descriptors. The m-th level decompression of the compressed data of the P 3D map descriptors yields the reconstructed data of the P 3D map descriptors, which includes: performing inverse quantization on the quantized data of each of the P 3D map descriptors to obtain P inverse quantized data; and obtaining the reconstructed data of each of the P 3D map descriptors based on the P inverse quantized data and the binarized data of each of the P 3D map descriptors.

[0026] In this embodiment, the j-th level retrieval can be the last level of a multi-level retrieval. The last level is based on the reconstructed data obtained from complete decompression, which can improve the accuracy of the retrieval.

[0027] In one possible design, S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. Based on the binarized data of the retrieved descriptors, a level i search is performed in the binarized data of the S 3D map descriptors to obtain P 3D map descriptors. This includes: based on the binarized data of the retrieved descriptors, a level i search is performed in the binarized data of the S representative 3D map descriptors to obtain at least one representative 3D map descriptor, and the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor are used as the P 3D map descriptors.

[0028] The 3D map descriptors in each dataset have a certain correlation or similarity. This represents the common or similar features of the 3D map descriptors in the corresponding dataset. Here, P can be greater than S. <T,S<T。

[0029] In this embodiment, by performing a first-level retrieval on a relatively small number of representative 3D map descriptors to improve the retrieval speed, and then performing a first-level or multi-level retrieval in the data set corresponding to the representative 3D map descriptors to improve the retrieval accuracy. Compared with retrieving in the reconstructed data of the 3D map obtained by complete decompression, the retrieval speed can be improved and the retrieval accuracy can be guaranteed.

[0030] In a possible design, Q < P < S < T. That is, the number of 3D map descriptors in the multi-level retrieval decreases gradually.

[0031] In a possible design, the method may further include: determining the magnitude relationship between the respective components of the retrieval descriptor and the corresponding components of the preset threshold vector according to the respective components of the retrieval descriptor and the corresponding components of the preset threshold vector. Performing a binary processing on the magnitude relationship to obtain the binary data of the retrieval descriptor.

[0032] In a possible design, determining the magnitude relationship between the respective components of the retrieval descriptor and the corresponding components of the preset threshold vector according to the respective components of the retrieval descriptor and the corresponding components of the preset threshold vector includes: respectively subtracting the respective components of the retrieval descriptor from the corresponding components of the preset threshold vector to obtain the difference of each component; determining the sign of each component according to the difference of each component. Performing a binary processing on the magnitude relationship to obtain the binary data of the retrieval descriptor includes: performing a binary processing on the signs of each component to obtain the binary data of the retrieval descriptor.

[0033] In a second aspect, an embodiment of the present application provides a method for retrieving a 3D map. The method may include: extracting the first binary data of S 3D map descriptors from the compressed data of S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in the 3D map. Performing an i-level retrieval in the first binary data of the S 3D map descriptors according to the first binary data of the retrieval descriptor to obtain P 3D map descriptors. The retrieval descriptor is a feature corresponding to the real environment extracted from the visual information collected by the sensor of the electronic device. Extracting the second binary data of P 3D map descriptors from the compressed data of P 3D map descriptors. Performing a j-level retrieval in the second binary data of the P 3D map descriptors according to the second binary data of the retrieval descriptor to obtain Q 3D map descriptors, where S, P, Q, i, and j are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrieval levels or the retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

[0034] As can be seen, in this embodiment, the first binarized data of S 3D map descriptors is extracted from the compressed data of S 3D map descriptors. Based on the first binarized data of the retrieved descriptors, a level i search is performed on the first binarized data of the S 3D map descriptors to filter out P 3D map descriptors. The second binarized data of P 3D map descriptors is extracted from the compressed data of the P 3D map descriptors. Based on the second binarized data of the retrieved descriptors, a level j search is performed on the second binarized data of the P 3D map descriptors to filter out a smaller number of 3D map descriptors. By extracting the binarized data of the 3D map descriptors retrieved at the first level from the compressed data and alternating with the first-level search, at least two levels of search are performed on the compressed 3D map to obtain the retrieved 3D map descriptors. The information content carried by the binarized data of the 3D map descriptors used in any two levels of search may differ, or the information completeness of the binarized data of the 3D map descriptors used in each level of search may differ. Compared to searching in the reconstructed data of a fully decompressed 3D map, this method can improve search speed and ensure search accuracy.

[0035] The first and second binarized data can be the binarized data used in any two levels of multi-level retrieval, and their binarization processing methods or lengths can be different.

[0036] In one possible design, N out of Q 3D map descriptors are used for localization. The 3D map points corresponding to the N 3D map descriptors are matched with the 3D map points corresponding to the search descriptors, where N is a positive integer, 0 < N ≤ Q. The method also includes: localization based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0037] In one possible design, both the i-th and j-th level searches employ a retrieval method based on a first distance. P 3D map descriptors belong to S 3D map descriptors. The position of the first binarized data of each of the P 3D map descriptors within the compressed data of the 3D map descriptors differs from the position of the second binarized data of the 3D map descriptors within the compressed data, where P < S. For example, this position could be a starting position or an ending position, etc.

[0038] It is evident that by employing at least two levels of differentiated retrieval, a comprehensive improvement in retrieval speed and accuracy can be achieved.

[0039] In one possible design, the length of the first binarized data of each of the P 3D map descriptors is less than the length of the second binarized data of each of the P 3D map descriptors.

[0040] It is evident that the shorter length of the first binarized data for each 3D map descriptor in the i-th level retrieval improves retrieval speed, while the longer length of the second binarized data for each 3D map descriptor in the j-th level retrieval improves retrieval accuracy. By employing at least two differentiated levels of retrieval, a comprehensive improvement in both retrieval speed and accuracy can be achieved.

[0041] In one possible design, the above method further includes: receiving the retrieval descriptor and binarizing the retrieval descriptor to obtain first binarized data and second binarized data of the retrieval descriptor. The execution entity (which may be an electronic device or a server (e.g., a cloud server)) performing the retrieval method described in the second aspect can receive retrieval descriptors sent by other devices. For example, the execution entity performing the retrieval method described in the second aspect receives retrieval descriptors collected and extracted by other electronic devices. The retrieval descriptor is then binarized to obtain first binarized data and second binarized data of the retrieval descriptor.

[0042] In one possible design, the above method further includes: receiving the visual information, extracting the retrieval descriptor from the visual information, and binarizing the retrieval descriptor to obtain first binarized data and second binarized data of the retrieval descriptor. The execution entity (which may be an electronic device or a server) performing the retrieval method described in the second aspect can receive visual information sent by other devices and extract the retrieval descriptor from it. The retrieval descriptor is then binarized to obtain first binarized data and second binarized data of the retrieval descriptor.

[0043] In one possible design, the above method further includes: responding to a user input operation to acquire visual information, triggering the sensor to acquire visual information from the real environment, obtaining the visual information, extracting the retrieval descriptor from the visual information, and binarizing the retrieval descriptor to obtain first and second binarized data of the retrieval descriptor. An execution entity (which may be an electronic device or a server) performing the retrieval method described in the second aspect can acquire the visual information and extract the retrieval descriptor from it. The retrieval descriptor is then binarized to obtain first and second binarized data of the retrieval descriptor.

[0044] In one possible design, the length of the first binarized data of the retrieval descriptor is equal to the length of the first binarized data of each of the S 3D map descriptors, and / or, the length of the second binarized data of the retrieval descriptor is equal to the length of the second binarized data of each of the S 3D map descriptors.

[0045] In this embodiment, by calculating the first distance of binarized data of equal length for retrieval in a 3D map, the retrieval speed can be improved.

[0046] In a possible design, the S 3D map descriptors are S representative 3D map descriptors. Each of the S representative 3D map descriptors corresponds to at least one data set, and each data set of the at least one data set includes at least one 3D map descriptor. The performing an i-level retrieval in the first binarized data of the S 3D map descriptors according to the first binarized data of the retrieval descriptor to obtain P 3D map descriptors includes: performing an i-level retrieval in the first binarized data of the S representative 3D map descriptors according to the first binarized data of the retrieval descriptor to obtain at least one representative 3D map descriptor; using the 3D map descriptors in the data sets corresponding to the at least one representative 3D map descriptor as the P 3D map descriptors.

[0047] In a possible design, Q < P < S < T. That is, the number of 3D map descriptors in the multi-level retrieval decreases gradually.

[0048] In a third aspect, an embodiment of the present application provides a method for retrieving a 3D map. The method may include: performing an m-level decompression on the compressed data of S 3D map descriptors to obtain first reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in the 3D map. Performing an i-level retrieval in the first reconstructed data of the S 3D map descriptors according to partial or all data of the retrieval descriptor to obtain P 3D map descriptors. The retrieval descriptor is a feature corresponding to the real environment extracted from the visual information collected by the sensor of the electronic device. Performing a k-level decompression on the compressed data of the P 3D map descriptors to obtain second reconstructed data of the P 3D map descriptors. The decompression process of the compressed data of the S 3D map descriptors includes the m-level decompression and the k-level decompression, or the decompression process of the compressed data of the S 3D map descriptors includes the k-level decompression, and the k-level decompression includes the m-level decompression. Performing a j-level retrieval in the second reconstructed data of the P 3D map descriptors according to partial or all data of the retrieval descriptor to obtain Q 3D map descriptors. S, P, Q, i, j, k, and m are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the total number of levels or the threshold of the retrieval levels of the 3D map retrieval, and L is a positive integer greater than 1.

[0049] As can be seen, in this embodiment, the compressed data of S 3D map descriptors is decompressed at level m to obtain the first reconstructed data of S 3D map descriptors. Based on partial or complete data of the retrieved descriptors, level i is performed in the first reconstructed data of the S 3D map descriptors to filter out P 3D map descriptors. The compressed data of P 3D map descriptors is decompressed at level k to obtain the second reconstructed data of P 3D map descriptors. Based on partial or complete data of the retrieved descriptors, level j is performed in the second reconstructed data of the P 3D map descriptors to filter out a smaller number of 3D map descriptors. The reconstructed data of the 3D map descriptors obtained through decompression at level one is alternated with the level one search. At least two levels of search involve staged decompression at level one, which can improve the search speed. The differentiated decompression settings of at least two levels of search can ensure the search accuracy. The degree of decompression or the degree of distortion of the reconstructed data of the 3D map descriptors used by any two levels of search are different. Compared to searching in the reconstructed data of a fully decompressed 3D map, this method can improve search speed and ensure search accuracy.

[0050] In one possible design, N out of Q 3D map descriptors are used for localization. The 3D map points corresponding to the N 3D map descriptors are matched with the 3D map points corresponding to the retrieval descriptor. N is a positive integer, 0 < N ≤ Q. The method further includes: performing localization based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0051] In this embodiment, when the 3D map points corresponding to the N retrieved 3D map descriptors match the map points corresponding to the retrieved descriptors, positioning can be performed based on the 3D map points corresponding to the N 3D map descriptors, which helps to improve positioning performance.

[0052] In one possible design, both the i-th level retrieval and the j-th level retrieval employ a retrieval method based on the second distance. The P 3D map descriptors belong to the S 3D map descriptors. The distortion degree of the first reconstructed data of each of the P 3D map descriptors is different from the distortion degree of the second reconstructed data of each of the P 3D map descriptors. The distortion degree of the first reconstructed data of each of the P 3D map descriptors is used to represent the degree of difference between the first reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor. The distortion degree of the second reconstructed data of each of the P 3D map descriptors is used to represent the degree of difference between the second reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor, where P < S.

[0053] It is evident that by employing at least two levels of differentiated retrieval, a comprehensive improvement in retrieval speed and accuracy can be achieved.

[0054] In one possible design, the distortion of the first reconstructed data of each of the P 3D map descriptors is greater than the distortion of the second reconstructed data of each of the P 3D map descriptors.

[0055] It is evident that the distortion of the first reconstructed data of each 3D map descriptor in the i-th level retrieval can be greater than the distortion of the second reconstructed data of each 3D map descriptor in the P 3D map descriptors. By ensuring that the distortion of the first-level retrieval is greater than that of the second-level retrieval, faster retrieval can be performed in the first-level retrieval to improve retrieval speed, while more refined retrieval can be performed in the second-level retrieval to improve retrieval accuracy.

[0056] In one possible design, the above method further includes: receiving the search descriptor and acquiring partial or all data of the search descriptor. The executing entity (which may be an electronic device or a server (e.g., a cloud server)) for performing the search method described in the third aspect can receive search descriptors sent by other devices. For example, the executing entity for performing the search method described in the third aspect receives search descriptors collected and extracted by other electronic devices. Then, partial or all data of the search descriptor is acquired according to the search requirements.

[0057] In one possible design, the above method further includes: receiving the visual information, extracting the retrieval descriptor from the visual information, and obtaining partial or all data of the retrieval descriptor. The executing entity (which may be an electronic device or a server) for performing the retrieval method described in the third aspect can receive visual information sent by other devices and extract the retrieval descriptor from it. Then, partial or all data of the retrieval descriptor is obtained according to the retrieval requirements.

[0058] In one possible design, the above method further includes: responding to a user-input visual information acquisition operation, triggering the sensor to acquire visual information from the real environment, obtaining the visual information, extracting the retrieval descriptor from the visual information, and acquiring partial or all data of the retrieval descriptor. The execution entity (which may be an electronic device or a server) used to perform the retrieval method described in the third aspect can acquire the visual information and extract the retrieval descriptor from it. Then, based on the retrieval requirements, partial or all data of the retrieval descriptor is acquired.

[0059] The 3D map retrieval method of this application embodiment can retrieve retrieval descriptors in different application scenarios in 3D maps, and has wide applicability.

[0060] In one possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one data set. Each of the at least one data set includes at least one 3D map descriptor. The step of performing an i-th level search on the first reconstructed data of the S 3D map descriptors based on partial or all data of the retrieved descriptors to obtain P 3D map descriptors includes: performing an i-th level search on the first reconstructed data of the S representative 3D map descriptors based on partial or all data of the retrieved descriptors to obtain at least one representative 3D map descriptor; and using the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0061] The 3D map descriptors in each dataset have a certain correlation or similarity. This represents the common or similar features of the 3D map descriptors in the corresponding dataset. Here, P can be greater than S. <T,S<T。

[0062] In this embodiment, a first-level search is performed using a smaller number of representative 3D map descriptors to improve search speed. Subsequently, first-level or multi-level searches can be performed on the datasets corresponding to these representative 3D map descriptors to improve search accuracy. Compared to searching through the reconstructed 3D map data obtained after complete decompression, this method improves search speed while ensuring search accuracy.

[0063] In one possible design, Q < P < S < T. That is, the number of 3D map descriptors decreases progressively in multi-level retrieval.

[0064] Fourthly, an embodiment of the present application provides a method for retrieving a 3D map. The method may include: decompressing the compressed data of S 3D map descriptors at the m-th level to obtain the reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in the 3D map. According to partial data or all data of the retrieval descriptor, perform an i-th level retrieval in the reconstructed data of the S 3D map descriptors to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to the real environment extracted from the visual information collected by the sensor of the electronic device. Extract the binary data of the P descriptors from the compressed data of the P 3D map descriptors. According to the binary data of the retrieval descriptor, perform a j-th level retrieval in the binary data of the P 3D map descriptors to obtain Q 3D map descriptors. S, P, Q, i, m, and j are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, where T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the total number of levels or the threshold of the retrieval levels of the 3D map retrieval, and L is a positive integer greater than 1.

[0065] It can be seen that in this embodiment, there is one level of retrieval for decompression in at least two levels of retrieval, which can improve the retrieval accuracy. There is one level of retrieval for extraction from compressed data in at least two levels of retrieval, which can improve the retrieval speed. Compared with retrieving in the reconstructed data of the 3D map obtained by complete decompression, the method for retrieving a 3D map provided by the embodiment of the present application can improve the retrieval speed and ensure the retrieval accuracy rate.

[0066] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor. N is a positive integer, 0 < N ≤ Q. The method further includes: performing positioning according to the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0067] In a possible design, the retrieval method used for the i-th level retrieval is a retrieval method based on the second distance, and the retrieval method used for the j-th level retrieval is a retrieval method based on the first distance.

[0068] In this embodiment, the retrieval method based on the second distance can improve the retrieval accuracy, and the retrieval method based on the first distance can improve the retrieval speed, thereby achieving a comprehensive improvement in retrieval accuracy and retrieval speed.

[0069] In one possible design, the above method further includes: receiving the search descriptor and binarizing the search descriptor to obtain binarized data of the search descriptor; the execution entity (which may be an electronic device or a server (e.g., a cloud server)) for performing the search method described in the fourth aspect can receive search descriptors sent by other devices. For example, the execution entity for performing the search method described in the first aspect receives search descriptors collected and extracted by other electronic devices. Then, the search descriptor is binarized to obtain binarized data of the search descriptor.

[0070] In one possible design, the above method further includes: receiving the visual information, extracting the retrieval descriptor from the visual information, and binarizing the retrieval descriptor to obtain binarized data of the retrieval descriptor; the execution entity (which may be an electronic device or a server) for performing the retrieval method described in the fourth aspect can receive visual information sent by other devices and extract the retrieval descriptor from it. Then, the retrieval descriptor is binarized to obtain binarized data of the retrieval descriptor.

[0071] In one possible design, the above method further includes: responding to a user-input visual information acquisition operation, triggering the sensor to acquire visual information from the real environment, obtaining the visual information, extracting the retrieval descriptor from the visual information, and binarizing the retrieval descriptor to obtain binarized data of the retrieval descriptor. The execution entity (which may be an electronic device or a server) performing the retrieval method described in the fourth aspect can receive visual information sent by other devices and extract the retrieval descriptor from it. The retrieval descriptor is then binarized to obtain binarized data of the retrieval descriptor.

[0072] In one possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. The step of performing an i-th level search on the reconstructed data of the S 3D map descriptors based on partial or all data of the retrieved descriptors to obtain P 3D map descriptors includes: performing an i-th level search on the reconstructed data of the S representative 3D map descriptors based on partial or all data of the retrieved descriptors to obtain at least one representative 3D map descriptor; and using the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0073] The 3D map descriptors in each dataset have a certain correlation or similarity. This represents the common or similar features of the 3D map descriptors in the corresponding dataset. Here, P can be greater than S. <T,S<T。

[0074] In this embodiment, a first-level search is performed using a smaller number of representative 3D map descriptors to improve search speed. Subsequently, first-level or multi-level searches can be performed on the datasets corresponding to these representative 3D map descriptors to improve search accuracy. Compared to searching through the reconstructed 3D map data obtained after complete decompression, this method improves search speed while ensuring search accuracy.

[0075] In one possible design, Q < P < S < T. That is, the number of 3D map descriptors decreases progressively in multi-level retrieval.

[0076] In combination with the first aspect or any possible design of the first aspect, or in combination with the second aspect or any possible design of the second aspect, or in combination with the third aspect or any possible design of the third aspect, or in combination with the fourth aspect or any possible design of the fourth aspect, the following possible designs also exist:

[0077] In one possible design, the retrieval descriptors include region descriptors or 3D map point descriptors. Correspondingly, the S 3D map descriptors can include S region descriptors or S 3D map point descriptors.

[0078] Fifthly, embodiments of this application provide a 3D map retrieval device. This device can be a chip or system-on-a-chip in an electronic device or server, or a functional module in an electronic device or server for implementing the method described in the first aspect or any possible implementation thereof. For example, the 3D map retrieval device includes: a retrieval module, an extraction module, and a decompression module.

[0079] In a possible design, an extraction module is configured to extract the binary data of S 3D map descriptors from the compressed data of S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in a 3D map; a retrieval module is configured to perform an i-level retrieval in the binary data of the S 3D map descriptors according to the binary data of a retrieval descriptor, so as to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to a real environment extracted from visual information collected by a sensor of an electronic device; a decompression module is configured to perform an m-level decompression on the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors; the decompression process of the compressed data of the P 3D map descriptors at least includes the m-level decompression; the retrieval module is configured to perform a j-level retrieval in the reconstructed data of the P 3D map descriptors according to partial data or all data of the retrieval descriptor, so as to obtain Q 3D map descriptors, where S, P, Q, i, j, and m are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrieval levels or the retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

[0080] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor, where N is a positive integer and 0 < N ≤ Q. The apparatus further includes: a pose determination module; the pose determination module is configured to perform positioning according to the 3D map points corresponding to the N 3D map descriptors, so as to obtain the pose information of the electronic device.

[0081] In a possible design, the retrieval method adopted for the i-level retrieval is a retrieval method based on a first distance, and the retrieval method adopted for the j-level retrieval is a retrieval method based on a second distance.

[0082] In a possible design, the apparatus further includes: an acquisition module, configured to: receive the retrieval descriptor and perform binary processing on the retrieval descriptor to obtain the binary data of the retrieval descriptor; or, receive the visual information, extract the retrieval descriptor from the visual information, and perform binary processing on the retrieval descriptor to obtain the binary data of the retrieval descriptor; or, in response to an operation of collecting visual information input by a user, trigger the sensor to collect visual information about the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and perform binary processing on the retrieval descriptor to obtain the binary data of the retrieval descriptor.

[0083] In a possible design, when N < Q, the decompression module is further configured to: perform k-level decompression on the compressed data of the Q 3D map descriptors to obtain the reconstructed data of the Q 3D map descriptors; perform r-level retrieval on the reconstructed data of the Q 3D map descriptors according to partial or all of the data of the retrieval descriptor to obtain the N 3D map descriptors; where N < Q < P, the decompression process of the compressed data of the P 3D map descriptors includes the m-level decompression and the k-level decompression, r and k are positive integers, m < k, j < r ≤ L.

[0084] In a possible design, the compressed data of the P 3D map descriptors includes the binarized data and the quantized data of each of the P 3D map descriptors, and the decompression module is specifically configured to: perform inverse quantization processing on the quantized data of each of the P 3D map descriptors to obtain P inverse quantization data, and the P inverse quantization data are used as the reconstructed data of the P 3D map descriptors; perform inverse quantization processing on the quantized data of each of the Q 3D map descriptors to obtain Q inverse quantization data; and obtain the reconstructed data of each of the Q 3D map descriptors according to the Q inverse quantization data and the binarized data of each of the Q 3D map descriptors.

[0085] In a possible design, when N = Q, the compressed data of the P 3D map descriptors includes the binarized data and the quantized data of each of the P 3D map descriptors, and the decompression module is specifically configured to: perform inverse quantization processing on the quantized data of each of the P 3D map descriptors to obtain P inverse quantization data; and obtain the reconstructed data of each of the P 3D map descriptors according to the P inverse quantization data and the binarized data of each of the P 3D map descriptors.

[0086] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors respectively corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. The retrieval module is configured to: perform i-level retrieval on the binarized data of the S representative 3D map descriptors according to the binarized data of the retrieval descriptor to obtain at least one representative 3D map descriptor; and use the 3D map descriptors in the data sets respectively corresponding to the at least one representative 3D map descriptor as the P 3D map descriptors.

[0087] In a sixth aspect, an embodiment of the present application provides a 3D map retrieval device, which can be a chip or a system-on-chip in an electronic device or a server, or can be a functional module in an electronic device or a server for implementing the method described in the first aspect or any possible implementation manner of the first aspect. For example, the 3D map retrieval device includes: a retrieval module, an extraction module, and a decompression module.

[0088] In a possible design, the extraction module is configured to extract first binary data of the S 3D map descriptors from the compressed data of the S 3D map descriptors, and the S 3D map descriptors correspond to multiple 3D map points in the 3D map; the retrieval module is configured to perform an i-level retrieval in the first binary data of the S 3D map descriptors according to the first binary data of the retrieval descriptor, so as to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to the real environment extracted from the visual information collected by the sensor of the electronic device; the extraction module is further configured to extract second binary data of the P 3D map descriptors from the compressed data of the P 3D map descriptors; the retrieval module is configured to perform a j-level retrieval in the second binary data of the P 3D map descriptors according to the second binary data of the retrieval descriptor, so as to obtain Q 3D map descriptors, where S, P, Q, i, and j are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of levels or the threshold of the retrieval levels of the 3D map retrieval, and L is a positive integer greater than 1.

[0089] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor, where N is a positive integer and 0 < N ≤ Q. The device further includes: a pose determination module; the pose determination module is configured to perform positioning according to the 3D map points corresponding to the N 3D map descriptors, so as to obtain the pose information of the electronic device.

[0090] In a possible design, the retrieval methods adopted for the i-level retrieval and the j-level retrieval are both retrieval based on the first distance. The P 3D map descriptors belong to the S 3D map descriptors, and the positions of the first binary data of each of the P 3D map descriptors in the compressed data of the 3D map descriptor are different from the positions of the second binary data of the 3D map descriptor in the compressed data of the 3D map descriptor, where P < S.

[0091] In one possible design, the length of the first binarized data of each of the P 3D map descriptors is less than the length of the second binarized data of each of the P 3D map descriptors.

[0092] In one possible design, the device further includes: an acquisition module, configured to: receive the retrieval descriptor and binarize the retrieval descriptor to obtain first binarized data and second binarized data of the retrieval descriptor; or, receive the visual information, extract the retrieval descriptor from the visual information, and binarize the retrieval descriptor to obtain first binarized data and second binarized data of the retrieval descriptor; or, in response to a user-input visual information acquisition operation, trigger the sensor to acquire visual information from the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and binarize the retrieval descriptor to obtain first binarized data and second binarized data of the retrieval descriptor.

[0093] In one possible design, the length of the first binarized data of the retrieval descriptor is equal to the length of the first binarized data of each of the S 3D map descriptors, and / or, the length of the second binarized data of the retrieval descriptor is equal to the length of the second binarized data of each of the S 3D map descriptors.

[0094] In one possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. The retrieval module is specifically used to: perform an i-th level retrieval on the first binarized data of the S representative 3D map descriptors based on the first binarized data of the retrieval descriptor to obtain at least one representative 3D map descriptor; and use the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0095] In a seventh aspect, embodiments of this application provide a 3D map retrieval device. This device can be a chip or system-on-a-chip in an electronic device or server, or a functional module in an electronic device or server for implementing the method described in the first aspect or any possible implementation thereof. For example, the 3D map retrieval device includes: a retrieval module, an extraction module, and a decompression module.

[0096] In a possible design, a decompression module is configured to perform m-level decompression on the compressed data of S 3D map descriptors to obtain first reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in a 3D map; a retrieval module is configured to perform i-level retrieval in the first reconstructed data of the S 3D map descriptors according to partial or all of the retrieval descriptor data to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to a real environment extracted from visual information collected by a sensor of an electronic device; the decompression module is further configured to perform k-level decompression on the compressed data of the P 3D map descriptors to obtain second reconstructed data of the P 3D map descriptors, the decompression process of the compressed data of the S 3D map descriptors includes the m-level decompression and the k-level decompression, or the decompression process of the compressed data of the S 3D map descriptors includes the k-level decompression, and the k-level decompression includes the m-level decompression; the retrieval module is configured to perform j-level retrieval in the second reconstructed data of the P 3D map descriptors according to partial or all of the retrieval descriptor to obtain Q 3D map descriptors, where S, P, Q, i, j, k, and m are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrieval levels or the retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

[0097] In a possible design, N of the Q 3D map descriptors are used for positioning, the 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor, N is a positive integer, 0 < N ≤ Q, and the device further includes: a pose determination module; the pose determination module is configured to perform positioning according to the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0098] In one possible design, both the i-th level retrieval and the j-th level retrieval employ a retrieval method based on the second distance. The P 3D map descriptors belong to the S 3D map descriptors. The distortion degree of the first reconstructed data of each of the P 3D map descriptors is different from the distortion degree of the second reconstructed data of each of the P 3D map descriptors. The distortion degree of the first reconstructed data of each of the P 3D map descriptors is used to represent the degree of difference between the first reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor. The distortion degree of the second reconstructed data of each of the P 3D map descriptors is used to represent the degree of difference between the second reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor, where P < S.

[0099] In one possible design, the distortion of the first reconstructed data of each of the P 3D map descriptors is greater than the distortion of the second reconstructed data of each of the P 3D map descriptors.

[0100] In one possible design, the device further includes: an acquisition module, configured to: receive the retrieval descriptor and acquire partial or all data of the retrieval descriptor; or, receive the visual information, extract the retrieval descriptor from the visual information, and acquire partial or all data of the retrieval descriptor; or, in response to a user-input visual information acquisition operation, trigger the sensor to acquire visual information from the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and acquire partial or all data of the retrieval descriptor.

[0101] In one possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. The retrieval module is specifically used to: perform a level i retrieval on the m-th reconstructed data of the S representative 3D map descriptors based on partial or all data of the retrieval descriptor to obtain at least one representative 3D map descriptor; and use the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0102] In an eighth aspect, an embodiment of the present application provides a 3D map retrieval device. This device can be a chip or a system-on-chip in an electronic device or a server, or can also be a functional module in an electronic device or a server for implementing the method described in the first aspect or any possible implementation manner of the first aspect. For example, the 3D map retrieval device includes: a retrieval module, an extraction module, and a decompression module.

[0103] In a possible design, the decompression module is used to perform the m-th level decompression on the compressed data of S 3D map descriptors to obtain the reconstructed data of the S 3D map descriptors. The S 3D map descriptors correspond to multiple 3D map points in the 3D map; the retrieval module is used to perform the i-th level retrieval in the reconstructed data of the S 3D map descriptors according to part or all of the data of the retrieval descriptor, so as to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to the real environment extracted from the visual information collected by the sensor of the electronic device; the extraction module is used to extract the binarized data of the P 3D map descriptors from the compressed data of the P 3D map descriptors; the retrieval module is further used to perform the j-th level retrieval in the binarized data of the P 3D map descriptors according to the binarized data of the retrieval descriptor, so as to obtain Q 3D map descriptors. S, P, Q, i, m, and j are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T. T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L. L represents the total number of levels or the threshold of the retrieval levels of the 3D map retrieval, and L is a positive integer greater than 1.

[0104] In a possible design, N of the Q 3D map descriptors are used for positioning. The 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor. N is a positive integer, 0 < N ≤ Q. The device further includes: a pose determination module; the pose determination module is used to perform positioning according to the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0105] In a possible design, the retrieval method used in the i-th level retrieval is a retrieval method based on the second distance, and the retrieval method used in the j-th level retrieval is a retrieval method based on the first distance.

[0106] In one possible design, the device further includes: an acquisition module, configured to: receive the retrieval descriptor and binarize the retrieval descriptor to obtain binarized data of the retrieval descriptor; or, receive the visual information, extract the retrieval descriptor from the visual information, and binarize the retrieval descriptor to obtain binarized data of the retrieval descriptor; or, in response to a user-input visual information acquisition operation, trigger the sensor to acquire visual information from the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and binarize the retrieval descriptor to obtain binarized data of the retrieval descriptor.

[0107] In one possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. The retrieval module is specifically used to: perform a level i retrieval on the m-th reconstructed data of the S representative 3D map descriptors based on partial or all data of the retrieval descriptor to obtain at least one representative 3D map descriptor; and use the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0108] Ninthly, embodiments of this application provide a 3D map retrieval device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described in any of the first to fourth aspects or any of the possible designs of the first to fourth aspects.

[0109] In a tenth aspect, embodiments of this application provide a computer-readable storage medium, characterized in that it includes a computer program, which, when executed on a computer, causes the computer to perform the method described in any one of the first to fourth aspects or any one of the possible designs of the first to fourth aspects.

[0110] Eleventhly, embodiments of this application provide a computer program or computer program product that, when executed on a computer, causes the computer to implement the method described in any of the first to fourth aspects and any possible implementation thereof.

[0111] It should be understood that the fifth to eleventh aspects of the embodiments of this application are consistent with the technical solutions of the first to fourth aspects of the embodiments of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be described again. Attached Figure Description

[0112] Figure 1 A schematic diagram of an application architecture provided in an embodiment of this application;

[0113] Figure 2 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application;

[0114] Figure 3 This is a schematic diagram of the structure of a server 30 provided in an embodiment of this application;

[0115] Figures 4a-4f Schematic diagrams illustrating various application scenarios provided in the embodiments of this application;

[0116] Figure 4g A schematic diagram of a user interface (e.g., a user interface for a 3D map application) displayed on an electronic device provided in an embodiment of this application;

[0117] Figures 5A to 5D This is a schematic diagram of a fragment of a 3D map retrieval method provided in an embodiment of this application;

[0118] Figure 6 This is a flowchart illustrating a 3D map retrieval method provided in an embodiment of this application;

[0119] Figure 7 This is a flowchart illustrating a 3D map retrieval method provided in an embodiment of this application;

[0120] Figure 8 This is a flowchart illustrating a 3D map retrieval method provided in an embodiment of this application;

[0121] Figure 9 This is a schematic diagram illustrating the processing steps of a 3D map retrieval method provided in an embodiment of this application.

[0122] Figure 10 A schematic diagram of the structure of a 3D map retrieval device provided in an embodiment of this application;

[0123] Figure 11 This is a schematic block diagram of a decoding device 1100 provided in an embodiment of this application. Detailed Implementation

[0124] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0125] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0126] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0127] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0128] The singular forms “a,” “said,” and “the” used in the embodiments of this application or the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. As used herein, the singular forms “a,” “some,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising,” “having,” “including,” and / or “containing,” as used herein, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0129] It should be noted that the terminology used in the embodiments of this application is for the purpose of describing specific embodiments only, and is not intended to limit this application.

[0130] Figure 1 This is a schematic diagram of an application architecture provided in an embodiment of this application, such as... Figure 1 As shown, the application architecture includes multiple electronic devices and servers, wherein the multiple electronic devices may include a first electronic device and one or more second electronic devices. Figure 1 (Taking two second electronic devices as an example), the one or more second electronic devices are several electronic devices other than the first electronic device. Communication can occur between the multiple electronic devices and the server, and between the multiple electronic devices themselves. For example, any device in the application architecture can communicate with other devices via wireless-fidelity (WiFi), Bluetooth, or 2 / 3 / 4 / 5 generation cellular communication. It should be understood that other communication methods, including future communication methods, can also be used between the server and the electronic devices, without specific limitations. It should be noted that in the embodiments of this application, "one or more second electronic devices" only refers to electronic devices other than the first electronic device, but does not limit whether the multiple electronic devices are of the same type.

[0131] The aforementioned electronic devices can be various types of devices equipped with cameras and display components. For example, electronic devices can be terminal devices such as mobile phones, tablets, laptops, and video recorders. Figure 1 (Taking a mobile phone as an example, electronic devices can also be devices used for virtual scene interaction, including VR glasses, AR devices, MR interactive devices, etc. Electronic devices can also be wearable electronic devices such as smartwatches and smart bracelets, and can also be devices mounted on vehicles, autonomous vehicles, drones, industrial robots, etc. The embodiments of this application do not specifically limit the specific form of electronic devices.)

[0132] In addition, the aforementioned electronic equipment may also be referred to as user equipment (UE), subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, terminal equipment, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handheld device, user agent, mobile client, client, or any other suitable term.

[0133] The aforementioned server can be one or more physical servers. Figure 1(Taking a physical server as an example) It can also be a computer cluster, a virtual machine or cloud server in a cloud computing scenario, and so on.

[0134] In this embodiment, the electronic device can install virtual scene applications (APPs) such as VR, AR, or MR applications, and can run VR, AR, or MR applications based on user operations (e.g., clicking, touching, swiping, shaking, voice control, etc.). The electronic device can collect visual information of any object in the environment through sensors, and then display virtual objects on the display component based on the collected visual information. These virtual objects can be virtual objects in VR, AR, or MR scenes (i.e., objects in the virtual environment).

[0135] In this embodiment, the electronic device can install navigation, detection, and control applications, and run the corresponding applications based on user input or preset programs. The electronic device can perform path planning, object detection, vehicle control, and other applications based on its own pose and other state information in the current environment.

[0136] Pose refers to the position and orientation information of an electronic device. It can be an absolute pose in the world coordinate system or a relative pose relative to a point in the environment.

[0137] The visual information involved in the embodiments of this application includes, but is not limited to, images and videos (without depth information) captured by a camera, images and videos with depth information captured by a depth sensor, data captured by LiDAR, and data captured by RaDAR.

[0138] It should be noted that, in the embodiments of this application, the virtual scene application in the electronic device can be an application built into the electronic device itself, or an application provided by a third-party service provider that the user installs himself, and there is no specific limitation on this.

[0139] In this embodiment, the electronic device may also be configured with a simultaneous localization and mapping (SLAM) system. The SLAM system can create a map in a completely unknown environment and use the map for localization, pose (position and attitude) determination, navigation, etc. In this embodiment, the map created by the SLAM system is referred to as a SLAM map. A SLAM map can be understood as a map drawn by the SLAM system based on environmental information collected by the acquisition device. The acquisition device may include a visual information acquisition device and an inertial measurement unit (IMU) in the electronic device. The visual information acquisition device may include, for example, a camera, depth camera, lidar, millimeter-wave radar, etc., and the IMU may include, for example, sensors such as gyroscopes and accelerometers.

[0140] In this embodiment, the SLAM map is also referred to as a 3D map. It should be noted that the 3D map includes, but is not limited to, the SLAM map, and may also include three-dimensional maps created using other technologies. This embodiment does not specifically limit the scope of the 3D map.

[0141] In one possible implementation, the 3D map may include multiple 3D map points, and correspondingly, the 3D map data may include data from these multiple 3D map points. 3D map points are points of interest or with salient features in the environment.

[0142] One possible way to acquire 3D map points is to use various devices such as LiDAR, aerial photography from a drone's perspective (oblique photography), high-definition panoramic cameras, and high-definition industrial cameras to capture images. These images are then extracted from the data captured by the aforementioned devices using methods such as ORB, scale-invariant feature transform (SIFT), accelerated up robust features (SURF), binary robust independent elementary features (BRIEF), binary robust invariant scalable keypoints (BRISK), fast retina keypoint (FREAK), D2Net, or a self-supervised training-based feature point detection and descriptor extraction method (SuperPoint).

[0143] 3D map point data can include:

[0144] (1) 3D map point descriptor

[0145] A 3D map point descriptor is a vector used to represent the local features of a corresponding 3D map point. In visual localization algorithms, 3D map point descriptors are used for matching between 3D map points. One possible approach is to calculate the distance between two 3D map point descriptors (which could be Euclidean distance, inner product distance, Hamming distance, etc.), and consider the two 3D map points as a match if the distance is less than a certain threshold.

[0146] (2) Spatial location of points on 3D map

[0147] The spatial location of 3D map points can be represented using X, Y, and Z coordinates on a three-dimensional spatial axis, or using latitude, longitude, altitude, or polar coordinates, etc. This application does not impose specific limitations on the method of representing the spatial location of 3D map points. The spatial location of a 3D map point can be its absolute position or its relative position. For example, with the center of the entire area as the origin, the spatial locations of all 3D map points are offsets relative to the origin.

[0148] In this embodiment, each 3D map point can be assigned a number and written into the 3D map data. Alternatively, the storage order of multiple 3D map points in memory can be used to implicitly represent the number of the 3D map points. It should be noted that the order among the multiple 3D map points contained in the 3D map has no practical meaning. Therefore, the aforementioned number can be considered as an identifier used to identify each 3D map point to distinguish them. However, this number is not used to limit the order among multiple 3D map points. For example, if the 3D map contains 3 3D map points, and their numbers are 1, 2, and 3, the processing of these 3 3D map points can be performed in the order of 1, 2, 3, or in the order of 3, 2, 1, or in the order of 2, 1, 3, and so on.

[0149] In one possible implementation, the 3D map data also includes multiple region descriptors. Any one of these region descriptors describes the features of some or all of the multiple 3D map points. That is, for any one region descriptor, it can describe the features of some or all of the multiple 3D map points, thus establishing a one-to-many relationship between region descriptors and 3D map points. Conversely, the features of each 3D map point can be described by some or all of the multiple region descriptors, again establishing a one-to-many relationship between 3D map points and region descriptors. Therefore, the relationship between multiple region descriptors and multiple 3D map points is many-to-many. Methods for generating region descriptors include, but are not limited to, traditional methods such as bag of words (BOW) and vector of locally aggregated descriptors (VLAD), as well as novel methods based on NetVLAD and artificial intelligence (AI). Similarly, multiple region descriptors can be identified by numbers to distinguish them, but these numbers do not restrict the order of the multiple region descriptors.

[0150] In one possible implementation, the 3D map data also includes the correspondence between 3D map points and descriptors, which explicitly describes which 3D map points correspond to any given descriptor, and which descriptors correspond to any given 3D map point.

[0151] Optionally, the above correspondence can be explicitly described using a correspondence table between the area descriptor numbers and the 3D map point numbers. For example, a 3D map contains 3 area descriptors, numbered T1 to T3, and 6 3D map points. The spatial location numbers of the 6 3D map points are P1 to P6, and the descriptor numbers of the 6 3D map points are F1 to F6. The correspondence table is shown in Table 1.

[0152] Table 1

[0153]

[0154] It should be noted that Table 1 is an example of a correspondence between the number of the area descriptor and the number of the 3D map point. This correspondence can also be presented in other formats or methods, and this application does not make any specific restrictions on it.

[0155] Optionally, the above correspondence can also be implicitly described using region descriptors and the storage locations of 3D map points. For example, T1 is stored in memory first, followed by the data of P1, P2 and P3, then T2 is stored, followed by the data of P2 and P3, and finally T3 is stored, followed by the data of P3, P4, P5 and P6.

[0156] Figure 2 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application, such as... Figure 2 As shown, electronic device 20 can be Figure 1 The illustrated embodiment includes at least one of a first electronic device and one or more second electronic devices. It should be understood that... Figure 2 The structure shown does not constitute a specific limitation on the electronic device 20. In other embodiments of this application, the electronic device 20 may include... Figure 2 The structure shown may have more or fewer components, or combine some components, or split some components, or have different component arrangements. Figure 2 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0157] Electronic device 20 may include: a chip 21, a memory 22 (one or more computer-readable storage media), a user interface 23, a display component 24, a camera 25, a sensor 26, a positioning module 27 for device positioning, and a transceiver 28 for communication. These components may communicate with each other via one or more buses 29.

[0158] Chip 21 may integrate one or more processors 211, a clock module 212, and a power management module 213. The clock module 212 integrated in chip 21 primarily provides the timer required for data transmission and timing control of processor 211, enabling clock functions for these operations. Processor 211 can perform calculations and generate operation control signals based on instruction opcodes and timing signals, thus controlling instruction fetching and execution. The power management module 213 integrated in chip 21 primarily provides a stable and highly accurate voltage for chip 21 and other components of electronic device 20.

[0159] Processor 211, also known as a central processing unit (CPU), may specifically include one or more processing units. For example, processor 211 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0160] In one possible implementation, the processor 211 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0161] The memory 22 can be connected to the processor 211 via bus 29, or coupled together with the processor 311, for storing various software programs and / or multiple sets of instructions. The memory 22 may include high-speed random access memory (e.g., cache memory) or non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 22 can store operating systems, such as Android, iOS, Windows, or Linux, or other embedded operating systems. The memory 22 can also store data, such as image data, point cloud data, 3D map data, pose data, coordinate system transformation information, map update information, etc. The memory 22 can also store computer-executable program code, including instructions, such as communication program instructions, SLAM system-related program instructions, etc. The memory 22 can also store one or more application programs, such as AR / VR / MR virtual scene applications, map applications, image management applications, navigation and control applications, etc. The memory 22 can also store user interface programs, which can display the content of the application, such as virtual objects in virtual scenes such as AR / VR / MR, in a realistic way through a graphical user interface and present it through the display component 24, and realize the user's control operation on the application through input controls such as menus, dialog boxes and buttons.

[0162] User interface 23 may be, for example, a touch panel that can detect user operation commands on it, or user interface 23 may be, for example, a keypad, physical buttons or a mouse.

[0163] Electronic device 20 may include one or more display components 24. Electronic device 20 can achieve display functions through the display components 24, the graphics processing unit (GPU) in chip 21, and the application processor (AP). The GPU is a microprocessor that performs image processing; it connects the display components 24 and the application processor, and performs mathematical and geometric calculations for graphics rendering. The display components 24 can display the interface content output by electronic device 20, such as images and videos from virtual scenes like AR / VR / MR. The interface content may include the interface of a running application and system-level menus, and can specifically consist of the following interface elements: input interface elements, such as buttons, text input boxes, scroll bars, and menus; and output interface elements, such as windows, labels, images, videos, and animations.

[0164] Display component 24 can be a display panel, lenses (e.g., VR glasses), projection screen, etc. The display panel can also be called a display screen, for example, it can be a touch screen, flexible screen, curved screen, etc., or other optical components. It should be understood that the display screen of the electronic device in the embodiments of this application can be a touch screen, flexible screen, curved screen, or other form of screen; that is, the display screen of the electronic device has the function of displaying images, and the specific material and shape of the display screen are not specifically limited.

[0165] For example, when the display component 24 includes a display panel, the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. Furthermore, in one possible implementation, the touch panel in the user interface 23 and the display panel in the display component 24 can be coupled together. For example, the touch panel can be located below the display panel, and the touch panel is used to detect the touch pressure applied to the display panel when the user inputs a touch operation (e.g., click, swipe, touch, etc.) through the display panel. The display panel is used for content display.

[0166] Camera 25 can be a monocular camera, a binocular camera, or a depth camera, used to capture / record the environment to obtain images / videos. The images / videos captured by camera 25 can be used as input data for a SLAM system, or displayed as images / videos via display component 24.

[0167] In one possible implementation, the camera 25 can also be regarded as a sensor. The images captured by the camera 25 can be in IMG format or other format types, and this application embodiment does not specifically limit them.

[0168] Sensor 26 can be used to collect data related to changes in the state of electronic device 20 (e.g., rotation, oscillation, movement, jitter, etc.). The data collected by sensor 26 can serve as input data for a SLAM system. Sensor 26 may include one or more sensors, such as an inertial measurement unit (IMU) or a time-of-flight (TOF) sensor. The IMU may include sensors such as a gyroscope and an accelerometer. The gyroscope measures the angular velocity of the electronic device during its movement, and the accelerometer measures the acceleration of the electronic device during its movement. The TOF sensor may include a light emitter and a light receiver. The light emitter emits light, such as laser light, infrared light, or radar waves, and the light receiver detects reflected light, such as reflected laser light, infrared light, or radar waves.

[0169] It should be noted that sensor 26 may also include other sensors, such as inertial sensors, barometers, magnetometers, wheel speedometers, etc., but this application embodiment does not specifically limit them.

[0170] The positioning module 27 is used to achieve physical positioning of the electronic device 20, for example, to obtain the initial position of the electronic device 20. The positioning module 27 may include one or more of a WiFi positioning module, a Bluetooth positioning module, a base station positioning module, and a satellite positioning module. The satellite positioning module may be equipped with a Global Navigation Satellite System (GNSS) to assist positioning; GNSS is not limited to the BeiDou system, the Global Positioning System (GPS) system, the GLONASS system, or the Galileo satellite navigation system.

[0171] Transceiver 28 is used to enable communication between electronic device 20 and other devices (e.g., servers, other electronic devices, etc.). Transceiver 28 integrates a transmitter and a receiver for transmitting and receiving radio frequency (RF) signals, respectively. In specific implementations, transceiver 28 includes, but is not limited to: an antenna system, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec (CODEC) chip, a subscriber identification module (SIM) card, and storage media, etc. In one possible implementation, transceiver 28 can also be implemented on a separate chip. Transceiver 28 supports at least one data network communication method among 2G / 3G / 4G / 5G, and / or supports at least one of the following short-range wireless communication methods: Bluetooth (BT) communication, Wireless Fidelity (WiFi) communication, Near Field Communication (NFC), Infrared (IR) wireless communication, Ultra Wide Bandwidth (UWB) communication, and ZigBee communication.

[0172] In this embodiment of the application, the processor 211 executes various functional applications and data processing of the electronic device 20 by running program code stored in the memory 22.

[0173] Figure 3 This is a schematic diagram of a server 30 provided in an embodiment of this application, such as... Figure 3 As shown, server 30 can be Figure 1 The server in the illustrated embodiment. Server 30 includes a processor 301, a memory 302 (one or more computer-readable storage media), and a transceiver 303. These components can communicate with each other via one or more buses 304.

[0174] Processor 301 can be one or more CPUs. If processor 301 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0175] The memory 302 can be connected to the processor 301 via bus 304, or it can be coupled together with the processor 301 to store various program codes and / or multiple sets of instructions, as well as data (e.g., map data, pose data, etc.). In specific implementations, the memory 302 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), etc.

[0176] The transceiver 303 mainly integrates a receiver and a transmitter, wherein the receiver is used to receive data (e.g., requests, images, etc.) sent by electronic devices, and the transmitter is used to send data (e.g., map data, pose data, etc.) to electronic devices.

[0177] It should be understood that Figure 3 The server 30 shown is only an example provided by the embodiments of this application. The server 30 may also have more components than shown, and the embodiments of this application do not specifically limit it.

[0178] In this embodiment, the processor 301 executes various functional applications and data processing of the server 30 by running program code stored in the memory 302.

[0179] The term "coupled" as used in the embodiments of this application means a direct connection or a connection through one or more intermediary components or circuits.

[0180] Figure 4a This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 4a As shown, this application scenario involves electronic devices collecting visual information through sensors, and combining the visual information with a 3D map from a server to determine the current pose of the electronic device.

[0181] The 3D map is provided by a server. The server creates the 3D map, compresses it, and transmits the compressed data to the electronic device. Upon receiving the compressed data, the electronic device decompresses it to obtain reconstructed 3D map data. This reconstructed data, combined with acquired visual information, determines the electronic device's current pose. Pose refers to the electronic device's position information; it can be an absolute pose in the world coordinate system or a relative pose relative to a point in the environment.

[0182] In this embodiment, the server can pre-create a 3D map, compress it, and store it locally, thus saving storage space. Alternatively, the server can also transmit the compressed 3D map data to other devices, such as cloud storage.

[0183] 1. The server creates a 3D map and compresses it to obtain compressed data of the 3D map, which is then stored locally.

[0184] The server can save local storage space by compressing 3D maps.

[0185] 2. The electronic device sends a map download request to the server. This map download request can be triggered in two ways:

[0186] (1) When a user opens a map application installed on their electronic device, the application uploads location information obtained based on GPS or WiFi positioning to its corresponding server. This upload operation can trigger a map download request. Since the uploaded content includes location information, the server can make a preliminary estimate based on the location information and transmit the compressed 3D map data of the area to which the location point indicated by the location information belongs to the electronic device. The range of the area to which the location point indicated by the location information belongs can be preset. For example, the area can be the administrative region at various levels where the location point is located (including counties, cities, countries, or administrative regions, etc.), or it can be a circular area with the location point as the center and a set distance as the radius.

[0187] (2) The user opens a map application installed on the electronic device and actively inputs or selects an area within the application. For example, the user actively inputs "xx business center" or selects "street A" from a list of "street A, street B, street C". The user's actions can trigger a map download request. Regardless of whether the user inputs or selects a geographical location, the server transmits compressed data of the 3D map of that location to the electronic device.

[0188] It should be understood that, in addition to the two methods described above, other methods can also be used to trigger the map download request in the embodiments of this application. For example, the electronic device can automatically detect whether the conditions for downloading a 3D map or starting the download of a 3D map are met, or the electronic device can start the download of a 3D map by detecting changes in ambient light or changes in the environment, so as to request the server to download a 3D map within a certain area, and the size of the area is not specifically limited.

[0189] 3. The server sends compressed data of the 3D map to the electronic device.

[0190] 4. Electronic devices collect visual information through sensors.

[0191] It should be noted that steps 3 and 4 are independent of each other and there is no requirement for their order.

[0192] 5. Electronic devices decompress the compressed data of the 3D map to obtain the reconstructed data of the 3D map.

[0193] 6. Electronic devices use visual information to locate themselves in a 3D map to obtain a pose corresponding to the visual information.

[0194] After receiving compressed 3D map data, electronic devices do not need to decompress it immediately. They only need to decompress it to obtain the reconstructed 3D map data before performing positioning based on visual information. For example, users can pre-download compressed 3D map data for a certain area as an "offline map," and only decompress the compressed 3D map data when positioning is required.

[0195] Figure 4b This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 4b As shown, this application scenario involves electronic devices collecting visual information through sensors, and the server combining the visual information from the electronic devices with a 3D map to determine the current pose of the electronic devices.

[0196] The 3D map is provided by the server. The server creates the 3D map, compresses it, and stores the compressed data locally. When visual information is received from the electronic device, the server decompresses the data to obtain reconstructed 3D map data. Combining the visual information with the reconstructed 3D map data, the server determines the current pose of the electronic device.

[0197] 1. The server creates a 3D map and compresses it to obtain compressed data of the 3D map, which is then stored locally.

[0198] 2. Electronic devices collect visual information through sensors.

[0199] 3. Electronic devices send visual information to the server.

[0200] 4. The server decompresses the compressed data of the 3D map to obtain the reconstructed data of the 3D map.

[0201] It should be understood that the server compresses the 3D map to save storage space.

[0202] 5. The server uses visual information to locate itself in the 3D map to obtain the pose corresponding to the visual information.

[0203] 6. The server sends poses to the electronic device.

[0204] Figure 4cThis is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 4c As shown, this application scenario involves electronic devices collecting visual information through sensors, and then combining the collected visual information with a 3D map to determine the current pose of the electronic device.

[0205] The 3D map is provided by an electronic device. The device creates the 3D map, compresses it, and stores the compressed data locally. When visual information is acquired, the electronic device decompresses the data to obtain reconstructed 3D map data. The current pose of the electronic device is determined by combining the acquired visual information with the reconstructed 3D map data.

[0206] 1. Electronic devices create 3D maps and compress them to obtain compressed data of the 3D map, which is then stored locally.

[0207] It should be understood that electronic devices compress 3D maps to save storage space.

[0208] 2. Electronic devices acquire visual information through sensors.

[0209] 3. Electronic devices decompress the compressed data of the 3D map to obtain the reconstructed data of the 3D map.

[0210] 4. Electronic devices use visual information to locate themselves in a 3D map to obtain a pose corresponding to the visual information.

[0211] Figure 4d This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 4b As shown, in this application scenario, the second electronic device collects visual information through sensors, and combines the visual information with a 3D map from the server to determine the current pose of the second electronic device.

[0212] The 3D map is created by a first electronic device. The first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed 3D map data to a server. The server then sends the compressed 3D map data to a second electronic device, which decompresses the data to obtain the reconstructed 3D map data. The current pose of the second electronic device is determined by combining the collected visual information and the reconstructed 3D map data.

[0213] In this embodiment of the application, the first electronic device can pre-create a 3D map, compress the 3D map, and then transmit it to the server, which can reduce the transmission bandwidth.

[0214] 1. The first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.

[0215] 2. The first electronic device sends compressed data of the 3D map to the server.

[0216] The first electronic device compresses the 3D map before transmitting it, which can reduce transmission bandwidth and improve transmission efficiency.

[0217] 3. The second electronic device sends a map download request to the server.

[0218] The second electronic device can also send map download requests based on... Figure 4a The triggering method is shown.

[0219] 4. The server sends compressed data of the 3D map to the second electronic device.

[0220] 5. The second electronic device decompresses the compressed data of the 3D map to obtain the reconstructed data of the 3D map.

[0221] 6. The second electronic device acquires visual information through sensors.

[0222] 7. The second electronic device locates itself in the 3D map based on the visual information to obtain the pose corresponding to the visual information.

[0223] Figure 4e This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 4c As shown, in this application scenario, the second electronic device collects visual information through sensors, and the server combines the visual information from the second electronic device with the 3D map from the first electronic device to determine the current pose of the second electronic device.

[0224] The 3D map is created by a first electronic device. The first electronic device creates the 3D map, compresses it, and then sends the compressed data to a server. The server decompresses the data to obtain the reconstructed 3D map data. Combining this data with visual information from a second electronic device, the server determines the current pose of the second electronic device.

[0225] 1. The first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map.

[0226] 2. The first electronic device sends compressed data of the 3D map to the server.

[0227] 3. The second electronic device acquires visual information through sensors.

[0228] 4. The second electronic device sends a location request to the server, which carries visual information.

[0229] 5. The server decompresses the compressed data of the 3D map to obtain the reconstructed data of the 3D map.

[0230] 6. The server uses visual information to locate itself in the 3D map to obtain the pose corresponding to the visual information.

[0231] 7. The server sends the position obtained from the location to the second electronic device.

[0232] Figure 4f This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 4d As shown, in this application scenario, the second electronic device collects visual information through sensors, and combines the visual information with a 3D map from the first electronic device to determine the current pose of the second electronic device.

[0233] The 3D map is created by a first electronic device. The first electronic device creates the 3D map, compresses the 3D map, and then sends the compressed 3D map data to a second electronic device. The second electronic device decompresses the data to obtain the reconstructed 3D map data. The current pose of the second electronic device is determined by combining the collected visual information and the 3D map from the first electronic device.

[0234] 1. The first electronic device creates a 3D map and compresses it to obtain compressed data of the 3D map, which is then stored locally.

[0235] 2. The second electronic device sends a map download request to the first electronic device.

[0236] 3. The first electronic device sends compressed data of the 3D map to the second electronic device.

[0237] 4. The second electronic device decompresses the compressed data of the 3D map to obtain the reconstructed data of the 3D map.

[0238] 5. The second electronic device acquires visual information through sensors.

[0239] 6. The second electronic device locates itself in the 3D map based on the visual information to obtain the pose corresponding to the visual information.

[0240] Figures 4a to 4f In the illustrated embodiment, the positioning algorithm used may include:

[0241] (1) Extract the descriptor of the region to be retrieved from the visual information. The algorithm used to extract the descriptor of the region to be retrieved is the same as the algorithm used to extract the region descriptor from the 3D map.

[0242] (2) Extract the 3D map points to be retrieved from the visual information, and obtain the spatial location and descriptor of the 3D map points to be retrieved. The algorithm for extracting the descriptor of the 3D map points to be retrieved is the same as the algorithm for extracting the descriptor of the 3D map points from the 3D map.

[0243] (3) Search among multiple region descriptors contained in the 3D map data based on the region descriptor to be searched to obtain multiple candidate region descriptors.

[0244] In this embodiment of the application, the distance between the descriptor of the region to be retrieved and each of the multiple region descriptors can be calculated. The distance can include Hamming distance, Manhattan distance or Euclidean distance, etc. Then, at least one region descriptor that meets the conditions (e.g., the distance is less than a threshold) is selected as the candidate region descriptor.

[0245] (4) Use the 3D map point descriptor to be retrieved to match the 3D map point descriptors corresponding to multiple candidate region descriptors respectively. Matching means calculating the similarity between the 3D map point descriptor to be retrieved and the 3D map point descriptors corresponding to multiple candidate region descriptors respectively, and finding the most similar 3D map point.

[0246] (5) Based on the matched 3D map points, the pose of the electronic device is calculated using pose estimation algorithms such as perspective-n-point (PnP) and efficient perspective-n-point camera poseestimation (EPnP).

[0247] exist Figures 4a to 4f In any application scenario, the electronic device's current pose is obtained by positioning based on the 3D map of this application embodiment. This pose can be applied to fields such as AR navigation, AR human-computer interaction, assisted driving, and autonomous driving. Taking AR navigation based on this pose as an example... Figure 4g This is a schematic diagram of a user interface displayed by an electronic device according to an embodiment of this application. The electronic device can display, based on this pose, such as... Figure 4g The user interface shown may include a navigation arrow indicating the way to meeting room 2. This navigation arrow may be a virtual object obtained from a server or locally based on the pose. The user interface may also include visual information acquired by sensors, such as... Figure 4g The building shown. The user refers to the electronic device as follows: Figure 4g The user interface shown leads to Meeting Room 2.

[0248] It should be noted that the reconstructed 3D map data obtained by decompression in the embodiments of this application can also be referred to as the reconstruction data of the 3D map.

[0249] and Figures 4a to 4fIn the embodiments shown, compression and decompression of the 3D map are all involved, as well as positioning based on the reconstructed data of the decompressed 3D map to obtain the current pose of the electronic device. For positioning, one possible approach is to perform positioning based on the reconstructed data of the fully decompressed 3D map. Another possible approach is to use the 3D map retrieval method provided in the following embodiments of this application to filter descriptors and use the descriptors for positioning. The 3D map retrieval method provided in the following embodiments of this application performs multi-level retrieval in the 3D map by alternating between retrieval and decompression or extraction from compressed data to obtain descriptors. Compared to retrieval in the reconstructed data of the fully decompressed 3D map, the 3D map retrieval method provided in the following embodiments of this application includes a non-fully decompressed retrieval in at least two levels of retrieval, which can improve retrieval speed, and the other level of retrieval in at least two levels can ensure retrieval accuracy. Non-fully decompressed refers to not fully decompressing all the reconstructed data of the 3D map. The 3D map retrieval method is described below.

[0250] The 3D map retrieval involved in this application embodiment refers to finding the most similar or most relevant 3D map points among multiple 3D map points.

[0251] The length of the binarized data involved in the embodiments of this application can be the number of bits in the binarized data (also called the number of bits). For example, if the binarized data is 110, then the number of bits in the binarized data is 3.

[0252] It should be noted that the retrieval method provided in this application embodiment can also be applied to other technical fields, such as image or video data retrieval, audio data retrieval, point cloud data retrieval, etc.

[0253] Based on the above description, the following provides a 3D map retrieval method according to embodiments of this application. For convenience, the various method embodiments described below are all described as a combination of a series of action steps. However, those skilled in the art should understand that the specific implementation of the technical solution of this application is not limited by the order of the described series of action steps.

[0254] See Figure 5A , Figure 5A This is a schematic diagram of a fragment of a 3D map retrieval method provided in an embodiment of this application. In some implementations, this method can be applied to... Figures 1 to 4f Any of the example electronic devices, or those that can also be applied to Figures 1 to 4f Any of the example servers. The method includes, but is not limited to, the following steps:

[0255] S101A: Extract the binarized data of S 3D map descriptors from the compressed data of S 3D map descriptors. The S 3D map descriptors correspond to multiple 3D map points in the 3D map.

[0256] The binarized data can be first binarized data, second binarized data, ..., nth binarized data, where n is a positive integer. It should be understood that the terms "first," "second," and "third" in the first, second, and third binarized data series do not have a specific order; they are merely used to distinguish them. The following explanation uses the nth binarized data as an example.

[0257] Extract the nth binarized data of S 3D map descriptors from the compressed data of S 3D map descriptors.

[0258] The nth binarized data of S 3D map descriptors is extracted from the compressed data of S 3D map descriptors. S is a positive integer. S represents the number of 3D map descriptors to be retrieved in the i-th level retrieval. S < T, where T represents the total number of 3D map descriptors in the 3D map.

[0259] The number of 3D map points corresponding to the S 3D map descriptors can be greater than S or equal to S.

[0260] For example, the S 3D map descriptors can be S region descriptors or S 3D map point descriptors. One region descriptor corresponds to multiple 3D map points, and one 3D map point descriptor corresponds to one 3D map point. For a detailed explanation, please refer to the explanations in the foregoing embodiments; they will not be repeated here.

[0261] The compressed data of S 3D map descriptors can be obtained by compressing S region descriptors or S 3D map point descriptors. This compression process can include compaction, which may include binarization and quantization. The nth binarized data of the S 3D map descriptors can be a portion of the compressed data of each of the S 3D map descriptors; for example, the nth binarized data of the S 3D map descriptors can be a portion or all of the binarized data of each of the S 3D map descriptors.

[0262] This binarization process includes, but is not limited to, iterative quantization (ITQ) hashing, locality-sensitive hashing (LSH), or spectral hashing. For example, binarization can map descriptors to Hamming space (also known as binary space) to obtain binarized data.

[0263] This quantization process includes, but is not limited to, scalar quantization, vector quantization, or product quantization. For example, quantization can compress a 3D map descriptor into one or more quantization indices, which are the quantized data of the 3D map descriptor. Each of the one or more quantization indices corresponds to a quantization center.

[0264] Optionally, the executing entity in this application embodiment can perform entropy decoding on the compressed data of S 3D map descriptors to extract the nth binarized data of the S 3D map descriptors.

[0265] S102A. Based on the binarized data of the search descriptors, perform the i-th level search in the binarized data of the S 3D map descriptors to obtain P 3D map descriptors.

[0266] Let's take the nth binary data as an example for explanation.

[0267] Based on the nth binarized data of the retrieval descriptor, perform the i-th level retrieval in the nth binarized data of the S 3D map descriptors to obtain P 3D map descriptors.

[0268] The search descriptor is a feature extracted from visual information collected by the sensors of an electronic device that corresponds to the real environment.

[0269] The retrieval descriptor here can be the descriptor of the region to be retrieved in the above embodiments, or the descriptor of the 3D map points to be retrieved. That is, the retrieval descriptor can be the descriptor of the region to be retrieved extracted from visual information, which is used to describe the features of some or all of the 3D map points among multiple 3D map points. The descriptor of the region to be retrieved can be a vector, for example, a 128-dimensional vector, or a 512-dimensional vector, etc. The dimension of the descriptor of the region to be retrieved is not limited to the examples above, and it can also be other values, which will not be illustrated in detail in the embodiments of this application. Alternatively, the retrieval descriptor can be the descriptor of the 3D map points to be retrieved extracted from visual information, which is used to represent the local features of the corresponding 3D map points. The descriptor of the 3D map points to be retrieved can be a vector, and the dimension of the vector can be any positive integer.

[0270] The methods for obtaining search descriptors may include, but are not limited to, the following. One possible implementation is that the execution entity in this embodiment can receive search descriptors sent by other devices. For example, the execution entity in this embodiment can be, for instance, […]. Figure 1 The first electronic device shown can receive a retrieval descriptor sent by the second electronic device. For example, the execution entity in this embodiment can be, for instance, a... Figure 1The server shown can receive retrieval descriptors sent by a first electronic device or a second electronic device. Alternatively, the execution entity in this embodiment can receive visual information sent by other devices and extract retrieval descriptors from the visual information. For example, the execution entity in this embodiment can be such as... Figure 1 The first electronic device shown can receive visual information sent by the second electronic device, and extract retrieval descriptors from the visual information. For example, the execution entity in this embodiment can be, for instance, a... Figure 1 The server shown can receive visual information sent by a first electronic device or a second electronic device, and extract retrieval descriptors from the visual information. In another possible implementation, the execution entity in this embodiment can detect a user-input visual information acquisition operation, and in response to this operation, trigger its own sensors to acquire visual information from the real environment, obtain visual information, and extract retrieval descriptors from the visual information. For example, the execution entity in this embodiment can be such as... Figure 1 The first electronic device shown, when detecting a user's input operation to collect visual information, responds to the operation by triggering its own sensors to collect visual information from the real environment, obtaining visual information, and extracting retrieval descriptors from the visual information. This visual information collection operation can be initiated by clicking, touching, swiping, or shaking, or by voice control or other means; this embodiment does not limit this. For example, after detecting a user's touch operation on the camera button on the user interface, the first electronic device's camera is activated in response to the touch operation, etc.

[0271] The nth binarized data of the retrieval descriptor can be part or all of the binarized data of the retrieval descriptor. As an example, the executing entity in this embodiment can perform binarization processing on the retrieval descriptor to obtain the binarized data of the retrieval descriptor, and select (also called extract) part or all of the binarized data of the retrieval descriptor as the nth binarized data of the retrieval descriptor.

[0272] The execution entity in this embodiment can use the nth binarized data of the retrieval descriptor to search the nth binarized data of the S 3D map descriptors to obtain the P 3D map descriptors that are most similar to or most relevant to the nth binarized data of the retrieval descriptor.

[0273] P is a positive integer, representing the number of 3D map descriptors selected by the i-th level of retrieval. P < T.

[0274] In some embodiments, P < S. Exemplarily, a set of 3D map descriptors is obtained through the i-th level of retrieval, and the number of 3D map descriptors in this set of 3D map descriptors is less than the number of 3D map descriptors in the set of 3D map descriptors before the i-th level of retrieval. In some embodiments, P may be greater than S. For a specific explanation, reference can be made to the following Figure 7 explanation of the illustrated embodiments, which will not be elaborated here.

[0275] When P < S, that is, P 3D map descriptors belong to S 3D map descriptors, the similarity or correlation degree between the n-th binary data of each of the P 3D map descriptors and the n-th binary data of the retrieval descriptor is higher than that of the other 3D map descriptors in the S 3D map descriptors except these P 3D map descriptors. Among them, there are many implementation manners for calculating the similarity or correlation degree. For example, by calculating the first distance between the n-th binary data of two 3D map descriptors (a 3D map descriptor and a retrieval descriptor), the correlation degree between the n-th binary data of the two 3D map descriptors is determined. Among them, the smaller the first distance, the higher the correlation degree.

[0276] In the embodiments of the present application, the first distance may include distances obtained using binary data such as Hamming distance. For example, the first distance may be Hamming distance. For another example, the first distance may be a distance obtained by taking the absolute value of the difference between the corresponding bits of two binary data and adding up all the absolute values of the differences. The retrieval based on the first distance refers to determining the correlation degree or similarity through calculating the first distance to screen 3D map descriptors.

[0277] S103A. Perform m-th level decompression on the compressed data of P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors.

[0278] The reconstructed data may be the first reconstructed data, the second reconstructed data,... the m-th reconstructed data, etc., where m is a positive integer. It should be understood that the "first", "second", and "third" in the first reconstructed data, the second reconstructed data, and the third reconstructed data, etc. have no sequence, and are only for distinguishing each other. Hereinafter, the m-th reconstructed data will be taken as an example for explanation.

[0279] Perform m-th level decompression on the compressed data of P 3D map descriptors to obtain the m-th reconstructed data of the P 3D map descriptors.

[0280] Through S101A and S102A, P 3D map descriptors can be screened out. Through S103A, the compressed data of these P 3D map descriptors can be decompressed at the m-th level to obtain the m-th reconstructed data of these P 3D map descriptors, so as to further perform retrieval based on the m-th reconstructed data. m is a positive integer. The m-th level decompression may include an inverse quantization process. The m-th level decompression may be a staged decompression (i.e., a part of the complete decompression), or it may also be a complete decompression.

[0281] The m-th reconstructed data of the P 3D map descriptors may be obtained by decompressing part or all of the compressed data of each of the P 3D map descriptors at the m-th level. For example, the m-th reconstructed data of the P 3D map descriptors may be inverse quantization data obtained by performing an inverse quantization process on part or all of the compressed data of each of the P 3D map descriptors. It should be noted that this inverse quantization data may also be referred to as reconstructed data, and this reconstructed data may be obtained by the inverse quantization method in the prior art. Another example is that the m-th reconstructed data of the P 3D map descriptors may be obtained by performing an inverse quantization process on the quantization data (a part of the compressed data) of each of the P 3D map descriptors to obtain P inverse quantization data, and then based on the P inverse quantization data and the binary data (a part of the compressed data) of each of the P 3D map descriptors.

[0282] In some embodiments, the m-th level decompression may further include other decompression processes such as entropy decoding and prediction, which will not be exemplified one by one in the embodiments of the present application.

[0283] S104A. According to part or all of the data of the retrieval descriptor, perform the j-th level retrieval among the reconstructed data of the P 3D map descriptors to obtain Q 3D map descriptors.

[0284] Q is a positive integer, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, where T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the total number of levels or the threshold of the retrieval levels of the 3D map retrieval, and L is a positive integer greater than 1.

[0285] This reconstructed data may be the first reconstructed data, the second reconstructed data,..., the m-th reconstructed data, etc., where m is a positive integer. It should be understood that the "first", "second", and "third" in the first reconstructed data, the second reconstructed data, and the third reconstructed data, etc. have no order of precedence, and are only for distinguishing each other. Hereinafter, the m-th reconstructed data will be taken as an example for explanation.

[0286] The partial or complete data of a retrieval descriptor refers to some or all of its components. A component of a retrieval descriptor is the value of one dimension of the retrieval descriptor vector. For example, the retrieval descriptor can be a vector, such as an M-dimensional vector, where each of the M dimensions is a component of the vector.

[0287] Similar to S102A, the execution entity in this embodiment can use partial or all of the data of the retrieval descriptor to search through the m-th reconstructed data of P 3D map descriptors to obtain Q 3D map descriptors that are most similar to or most relevant to the m-th reconstructed data of the retrieval descriptor. Q can be less than P. Alternatively, it can be understood that another set of 3D map descriptors is obtained through the j-th level retrieval, and the number of 3D map descriptors in this set is less than the number of 3D map descriptors obtained through the i-th level retrieval.

[0288] The similarity or relevance between the m-th reconstructed data of each of the Q 3D map descriptors and some or all of the data of the retrieved descriptor is higher than that of the other 3D map descriptors among the P 3D map descriptors excluding the Q 3D map descriptors. There are many ways to calculate the similarity or relevance. For example, the relevance between two 3D map descriptors can be determined by calculating the second distance between the m-th reconstructed data of the 3D map descriptor and some or all of the data of the retrieved descriptor. The second distance can include, but is not limited to, Euclidean distance, inner product distance, cosine distance, Manhattan distance, etc. For example, the smaller the Euclidean distance, the higher the relevance; the larger the inner product distance, the higher the relevance.

[0289] It is understandable that a condition judgment step can be executed after S104A. If the condition is met, the search in this embodiment can end; if the condition is not met, the search can continue at one or more levels. This condition can be arbitrary and can be reasonably set according to needs. For example, the condition can be that the similarity or relevance is higher than or equal to a preset threshold, or that the number of retrieved 3D map descriptors is less than or equal to a preset number. Another example is that the condition can be that the following N 3D map descriptors used for positioning are retrieved.

[0290] The 3D map retrieval method of this application embodiment is a segmented process. Other levels of retrieval may be included before and / or after this segmented process. In other words, the 3D map retrieval method may include more than... Figure 5A This demonstrates multi-level retrieval with more levels, such as 3, 4, or 5 levels, etc. For example, as shown... Figure 5A The illustrated embodiment may have more levels of retrieval before S101A, or, as... Figure 5A Further levels of retrieval can be performed after S104A in the illustrated embodiment, which will not be illustrated here.

[0291] In some embodiments, the Q 3D map descriptors obtained through multi-level retrieval can be used for localization. If the 3D map points corresponding to N of the Q 3D map descriptors match the 3D map points corresponding to the retrieved descriptors, localization can be performed based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device. This electronic device can be an electronic device that acquires the aforementioned visual information. 0 < N ≤ Q.

[0292] In this embodiment, S binarized data of S 3D map descriptors are extracted from compressed data of S 3D map descriptors. Based on the binarized data of the search descriptors, a level i search is performed on the binarized data of the S 3D map descriptors to filter out P 3D map descriptors. The compressed data of the P 3D map descriptors is then decompressed at level m to obtain reconstructed data of P 3D map descriptors. Based on partial or complete data of the search descriptors, a level j search is performed on the reconstructed data of the P 3D map descriptors to filter out a smaller number of 3D map descriptors. At least one level of the search uses the binarized data from the compressed data, which can improve the search speed; the other level uses the decompressed reconstructed data to ensure search accuracy. Compared to searching in the fully decompressed reconstructed 3D map data, the 3D map retrieval method provided in this embodiment can improve search performance.

[0293] See Figure 5B , Figure 5B This is a schematic diagram of a fragment of a 3D map retrieval method provided in an embodiment of this application. In some implementations, this method can be applied to... Figures 1 to 4f Any of the example electronic devices, or those that can also be applied to Figures 1 to 4f Any of the example servers. The method includes, but is not limited to, the following steps:

[0294] S101B: Extract the first binarized data of S 3D map descriptors from the compressed data of S 3D map descriptors. The S 3D map descriptors correspond to multiple 3D map points in the 3D map.

[0295] The term "first binarized data" can refer to any one of the following: first binarized data, second binarized data, ..., nth binarized data, where n is a positive integer. It should be understood that the terms "first," "second," and "third" in the context of first, second, and third binarized data do not imply any order; they are merely used to distinguish them. The following explanation uses the nth binarized data as an example.

[0296] Extract the nth binary data of S 3D map descriptors from the compressed data of S 3D map descriptors. The S 3D map descriptors correspond to multiple 3D map points in the 3D map.

[0297] S102B. Based on the first binarized data of the retrieval descriptors, perform the i-th level retrieval in the first binarized data of the S 3D map descriptors to obtain P 3D map descriptors.

[0298] The following explanation uses the nth binarized data as an example. Based on the nth binarized data of the search descriptor, the i-th level search can be performed on the nth binarized data of the S 3D map descriptors to obtain P 3D map descriptors.

[0299] For detailed explanations of S101B and S102B, please refer to [link / reference needed]. Figure 5A S101A and S102A of the illustrated embodiment.

[0300] S103B: Extract the second binarized data of P 3D map descriptors from the compressed data of P 3D map descriptors.

[0301] The second binarized data can refer to any one of the second, third, ..., q-th binarized data, where q is a positive integer. It should be understood that the terms "second" and "third" in "second binarized data" and "third binarized data," etc., are not sequential but merely distinguishable. The following explanation uses the q-th binarized data as an example. One example: q = n + 1.

[0302] S101B and S102B can filter out P 3D map descriptors from S 3D map descriptors, and then extract the q-th binarized data of the P 3D map descriptors from their compressed data for further retrieval. The q-th binarized data of the P 3D map descriptors can be part or all of the compressed data of each of the P 3D map descriptors. For example, the q-th binarized data of the P 3D map descriptors can be part or all of the binarized data of each of the P 3D map descriptors.

[0303] The n-th binary data of each of the S 3D map descriptors is different from the q-th binary data of each of the P 3D map descriptors. In one implementation, this difference can be in length, that is, the length of the n-th binary data of each of the S 3D map descriptors is different from the length of each of the P 3D map descriptors. In another implementation, this difference can be in the binary processing method corresponding to the n-th binary data and the q-th binary data, that is, the binary processing method corresponding to the n-th binary data of each of the S 3D map descriptors is different from the binary processing method corresponding to each of the P 3D map descriptors. For example, the n-th binary data of each of the S 3D map descriptors is obtained by a binary processing method (e.g., locality-sensitive hashing), and each of the P 3D map descriptors is obtained by another binary processing method (e.g., spectral hashing).

[0304] The length of the n-th binary data can be measured or represented by the number of bits of the n-th binary data. Similarly, the length of the q-th binary data can be measured or represented by the number of bits of the q-th binary data.

[0305] When the P 3D map descriptors belong to the S 3D map descriptors (P < S), the position of the n-th binary data of each of the P 3D map descriptors in the compressed data of the 3D map descriptor is different from the position of the q-th binary data of this 3D map descriptor in the compressed data of this 3D map descriptor. This position can be the starting position or the ending position, etc.

[0306] In one example, the component of the original 3D map descriptor corresponding to the n-th binary data of each of the P 3D map descriptors is different from the component of the original 3D map descriptor corresponding to the q-th binary data of the corresponding 3D map descriptor. That is, the n-th binary data of a 3D map descriptor can be the binary data of the first part of this 3D map descriptor (e.g., the components of the first 128 dimensions of the 3D map descriptor), and the q-th binary data of this 3D map descriptor can be the binary data of the second part of this 3D map descriptor (e.g., the components of the last 128 dimensions of the 3D map descriptor).

[0307] When P 3D map descriptors belong to S 3D map descriptors (P < S), the length of the n-th binary data of each 3D map descriptor among the P 3D map descriptors can be less than the length of the q-th binary data of each 3D map descriptor among the P 3D map descriptors. The shorter length of the n-th binary data of each 3D map descriptor in the i-th level of retrieval can improve the retrieval speed, and the longer length of the q-th binary data of the 3D map descriptor in the j-th level of retrieval can improve the retrieval accuracy. Through at least two levels of differential retrieval, the comprehensive improvement of retrieval speed and retrieval accuracy can be achieved. Or, the length of the n-th binary data of each 3D map descriptor among the P 3D map descriptors can be greater than the length of the q-th binary data of each 3D map descriptor among the P 3D map descriptors. Or, the length of the n-th binary data of each 3D map descriptor among the P 3D map descriptors can be equal to the length of the q-th binary data of each 3D map descriptor among the P 3D map descriptors.

[0308] S104B. Perform the j-th level of retrieval in the second binary data of the P 3D map descriptors according to the second binary data of the retrieval descriptor to obtain Q 3D map descriptors.

[0309] The following takes the q-th binary data as an example for explanation.

[0310] Perform the j-th level of retrieval in the q-th binary data of the P 3D map descriptors according to the q-th binary data of the retrieval descriptor to obtain Q 3D map descriptors.

[0311] S, P, Q, i, and j are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, where T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the total number of levels of retrieval or the retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

[0312] The q-th binary data of the retrieval descriptor can be part or all of the binary data of the retrieval descriptor. In one example, the execution entity of this embodiment can perform binary processing on the retrieval descriptor to obtain the binary data of the retrieval descriptor, and select (also referred to as extract) part or all of it as the (n + 1)-th representation of the retrieval descriptor. It should be noted that the n-th binary data of the retrieval descriptor and the q-th binary data of the retrieval descriptor can be the same or different.

[0313] Similar to S102B, the execution entity in this embodiment can use the q-th binarized data of the retrieval descriptor to search through the q-th binarized data of P 3D map descriptors to obtain the Q 3D map descriptors that are most similar to or most relevant to the q-th binarized data of the retrieval descriptor. Q can be less than P. Alternatively, it can be understood that another set of 3D map descriptors is obtained through the j-th level retrieval, and the number of 3D map descriptors in this set is less than the number of 3D map descriptors obtained through the i-th level retrieval.

[0314] The similarity or relevance between the q-th binarized data of each of the Q 3D map descriptors and the q-th binarized data of the retrieved descriptor is higher than that of the other 3D map descriptors among the P 3D map descriptors excluding the Q 3D map descriptors. For an explanation of the calculation method for similarity or relevance, please refer to the relevant explanation in S102B, which will not be repeated here.

[0315] Optionally, the retrieval methods used in the i-th level retrieval in S102B and the j-th level retrieval in S104B are both based on the first distance. For a detailed explanation, please refer to [link to relevant documentation]. Figure 5A The explanation of S102A in the illustrated embodiment will not be repeated here.

[0316] It is understandable that a condition judgment step can be executed after S104B. If the condition is met, the search in this embodiment can be terminated; if the condition is not met, the search can continue to be performed at one or more levels. This condition can be any condition, which can be reasonably set according to needs. For example, the condition can be that the similarity or relevance is higher than or equal to a preset threshold, or that the number of 3D map descriptors obtained is less than or equal to a preset number, etc. This application embodiment will not provide examples of each of these conditions.

[0317] The 3D map retrieval method of this application embodiment is a segmented process. Other levels of retrieval may be included before and / or after this segmented process. In other words, the 3D map retrieval method may include more than... Figure 5B This demonstrates multi-level retrieval with more levels, such as 3, 4, or 5 levels, etc. For example, as shown... Figure 5B The illustrated embodiment may have more levels of retrieval before S101B, or, as... Figure 5B Further levels of retrieval can be performed after S104B in the illustrated embodiment, which will not be illustrated here.

[0318] In some embodiments, the Q 3D map descriptors obtained through multi-level retrieval can be used for localization. If the 3D map points corresponding to N of the Q 3D map descriptors match the 3D map points corresponding to the retrieved descriptors, localization can be performed based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device. This electronic device can be an electronic device that acquires the aforementioned visual information. 0 < N ≤ Q.

[0319] In this embodiment, first binarized data of S 3D map descriptors is extracted from compressed data of S 3D map descriptors. Based on the first binarized data of the retrieved descriptors, a level i search is performed on the first binarized data of the S 3D map descriptors to filter out P 3D map descriptors. Second binarized data of P 3D map descriptors is extracted from compressed data of P 3D map descriptors. Based on the second binarized data of the retrieved descriptors, a level j search is performed on the second binarized data of the P 3D map descriptors to filter out a smaller number of 3D map descriptors. By extracting the binarized data of the 3D map descriptors retrieved at the first level from the compressed data and alternating with the first-level search, at least two levels of search are performed on the compressed 3D map to obtain the retrieved 3D map descriptors. The information content carried by the binarized data of the 3D map descriptors used in any two levels of search may be different, or the information completeness of the binarized data of the 3D map descriptors used in each level of search may be different. Compared to searching in the reconstructed data of a fully decompressed 3D map, this method can improve search speed and ensure search accuracy.

[0320] See Figure 5C , Figure 5C This is a schematic diagram of a fragment of a 3D map retrieval method provided in an embodiment of this application. In some implementations, this method can be applied to... Figures 1 to 4f Any of the example electronic devices, or those that can also be applied to Figures 1 to 4f Any of the example servers. The method includes, but is not limited to, the following steps:

[0321] S101C. Perform m-th level decompression on the compressed data of S 3D map descriptors to obtain the first reconstructed data of S 3D map descriptors. The S 3D map descriptors correspond to multiple 3D map points in the 3D map.

[0322] The first reconstructed data can refer to any one of the first reconstructed data, the second reconstructed data, ..., the m-th reconstructed data, where m is a positive integer. It should be understood that the terms "first," "second," and "third" in the first, second, and third reconstructed data are not sequential; they are merely used to distinguish them. The following explanation uses the example of the first reconstructed data being the m-th reconstructed data.

[0323] The compressed data of S 3D map descriptors is decompressed at the m-th level to obtain the m-th reconstructed data of S 3D map descriptors. The S 3D map descriptors correspond to multiple 3D map points in the 3D map.

[0324] S and m are positive integers. S represents the number of 3D map descriptors to be retrieved in the i-th level of retrieval. S < T, where T represents the total number of 3D map descriptors in the 3D map.

[0325] The number of 3D map points corresponding to the S 3D map descriptors can be greater than S or equal to S.

[0326] The compressed data of S 3D map descriptors can be obtained by compressing S region descriptors or S 3D map point descriptors. The compression process may include quantization. Optionally, the compression process may also include at least one of the following: binarization, prediction, or entropy coding.

[0327] The m-th reconstructed data of S 3D map descriptors is obtained by performing m-th level decompression on a portion of the compressed data of each of the S 3D map descriptors.

[0328] m is a positive integer. The m-th level of decompression may include dequantization. The m-th level of decompression may be a staged decompression (i.e., part of a complete decompression).

[0329] The m-th reconstructed data of S 3D map descriptors can be obtained by decompressing a portion of the compressed data of each of the S 3D map descriptors at the m-th level. For example, the m-th reconstructed data of S 3D map descriptors can be obtained by performing an inverse quantization process on a portion of the compressed data of each of the S 3D map descriptors. It should be noted that this inverse quantization data can also be called reconstructed data, and this reconstructed data can be obtained through an inverse quantization method in existing technology. As another example, the m-th reconstructed data of S 3D map descriptors can be obtained by performing inverse quantization on a portion of the quantized data (a portion of the compressed data) of each of the S 3D map descriptors to obtain S inverse quantized data, and then obtaining the data based on the S inverse quantized data and a portion of the binarized data (a portion of the compressed data) of each of the S 3D map descriptors.

[0330] In some embodiments, the m-th level decompression may also include other decompression processes such as entropy decoding and prediction, which will not be illustrated in detail in the embodiments of this application.

[0331] S102C. Based on partial or complete data of the retrieved descriptors, perform the i-th level retrieval in the first reconstructed data of the S 3D map descriptors to obtain P 3D map descriptors.

[0332] The following takes the first reconstructed data being the m-th reconstructed data as an example for explanation.

[0333] Perform the i-th level of retrieval in the m-th reconstructed data of the S 3D map descriptors according to part or all of the data of the retrieval descriptor, so as to obtain P 3D map descriptors.

[0334] Part or all of the data of the retrieval descriptor refers to part or all of the components of the retrieval descriptor. A component of the retrieval descriptor refers to the value of one dimension in this vector of the retrieval descriptor. For example, the retrieval descriptor can be a vector, for example, a vector of M dimensions, and each dimension in the M dimensions is a component of this vector.

[0335] The execution entity of this embodiment can use part or all of the data of the retrieval descriptor to perform retrieval in the m-th reconstructed data of the S 3D map descriptors, so as to obtain P 3D map descriptors that are most similar or relevant to the m-th reconstructed data of the retrieval descriptor.

[0336] In some embodiments, P < S. Exemplarily, a set of 3D map descriptors is obtained through the i-th level of retrieval, and the number of 3D map descriptors in this set of 3D map descriptors is less than the number of 3D map descriptors in the set of 3D map descriptors before the i-th level of retrieval. In some embodiments, P can be greater than S. For specific explanations, reference can be made to the explanations of the embodiments shown below Figure 7 and will not be elaborated here.

[0337] When P < S, that is, the P 3D map descriptors belong to the S 3D map descriptors, the similarity or relevance between the m-th reconstructed data of each of the P 3D map descriptors and part or all of the data of the retrieval descriptor is higher than that of the other 3D map descriptors among the S 3D map descriptors except these P 3D map descriptors. Among them, there are many implementation manners for the calculation method of similarity or relevance. For example, the relevance between two 3D map descriptors (the k-th reconstructed data of the 3D map descriptor and part or all of the data of the retrieval descriptor) is determined by calculating the second distance. The second distance can include but is not limited to Euclidean distance, inner product distance, cosine distance, Manhattan distance, etc. For example, the smaller the Euclidean distance, the higher the relevance, and the larger the inner product distance, the higher the relevance.

[0338] S103C. Perform the k-th level of decompression on the compressed data of the P 3D map descriptors to obtain the second reconstructed data of the P 3D map descriptors. The decompression process of the compressed data of the S 3D map descriptors includes the m-th level of decompression and the k-th level of decompression, or the decompression process of the compressed data of the S 3D map descriptors includes the k-th level of decompression, and the k-th level of decompression includes the m-th level of decompression.

[0339] The second reconstructed data may refer to any one of the second reconstructed data, the third reconstructed data, …, the k-th reconstructed data, etc., where k is a positive integer and m < k. It should be understood that the "second" and "third" in the second reconstructed data and the third reconstructed data, etc. have no order of precedence, and are only for distinguishing each other. Hereinafter, an example where the second reconstructed data is the k-th reconstructed data will be used for explanation.

[0340] Perform the k-th level decompression on the compressed data of P 3D map descriptors to obtain the k-th reconstructed data of P 3D map descriptors. The decompression process of the compressed data of S 3D map descriptors includes the m-th level decompression and the k-th level decompression, or the decompression process of the compressed data of S 3D map descriptors includes the k-th level decompression, and the k-th level decompression includes the m-th level decompression.

[0341] Through S101C and S102C, P 3D map descriptors can be screened out from S 3D map descriptors, and then the k-th level decompression is performed on the compressed data of these P 3D map descriptors to obtain the k-th reconstructed data of P 3D map descriptors, so as to further perform retrieval based on the k-th reconstructed data. In some embodiments, k = m + 1.

[0342] The k-th reconstructed data of P 3D map descriptors can be obtained by performing the k-th level decompression on part or all of the compressed data of each of the P 3D map descriptors.

[0343] In some embodiments, the retrieval methods used in the i-th level retrieval and the j-th level retrieval are both retrieval methods based on the second distance. P 3D map descriptors belong to S 3D map descriptors. The distortion degree of the m-th reconstructed data of each of the P 3D map descriptors is different from the distortion degree of the k-th reconstructed data of each of the P 3D map descriptors. The distortion degree of the m-th reconstructed data of each of the P 3D map descriptors is used to represent the difference degree between the m-th reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor, and the distortion degree of the k-th reconstructed data of each of the P 3D map descriptors is used to represent the difference degree between the k-th reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor, where P < S.

[0344] In some embodiments, the distortion degree of the m-th reconstructed data of each of the P 3D map descriptors is greater than the distortion degree of the k-th reconstructed data of each of the P 3D map descriptors. The result of the subsequent level of retrieval in at least two levels of retrieval has a smaller distortion degree relative to the result of the previous level of retrieval, thereby improving the accuracy of the retrieval result.

[0345] For example, P 3D map descriptors belong to S 3D map descriptors. Taking one 3D map descriptor in the P 3D map descriptors, with m=1 and k=2, as an example, the compressed data of this 3D map descriptor includes first quantized data and second quantized data. The first reconstructed data of this 3D map descriptor is obtained by performing a first-level decompression on the first quantized data. The first reconstructed data of this 3D map descriptor may include the reconstructed data of some components of the original vector of this 3D map descriptor. The second reconstructed data of this 3D map descriptor is obtained by performing a second-level decompression on the second quantized data. The second reconstructed data of this 3D map descriptor may include the reconstructed data of another part of the original vector of this 3D map descriptor. That is, the decompression process of the compressed data of this 3D map descriptor includes a first (m=1) level decompression and a second (k=2) level decompression.

[0346] For example, P 3D map descriptors belong to S 3D map descriptors. Taking one 3D map descriptor from the P 3D map descriptors, with m=1 and k=2, as an example, the compressed data of this 3D map descriptor includes first quantized data and second quantized data. Unlike the previous example, the second quantized data here can be obtained by quantizing the residual of the first quantized data. The first reconstructed data of this 3D map descriptor is obtained by first-level decompression of the first quantized data. The first reconstructed data of this 3D map descriptor can include the first reconstructed data of the original vector of the 3D map descriptor (with higher precision). The second reconstructed data of the 3D map descriptor is obtained by performing a second-level decompression on the second quantized data. The second reconstructed data of the 3D map descriptor may include the reconstructed data of the residual data of the 3D map descriptor mentioned above. The first reconstructed data and the second reconstructed data of the 3D map descriptor are added together to obtain the reconstructed data of the original vector of the 3D map descriptor (reconstructed data with higher precision). That is, the decompression process of the compressed data of the 3D map descriptor includes a second (k=2) level decompression, and the second (k=2) level decompression includes a first (m=1) level decompression.

[0347] S104C. Based on partial or complete data of the retrieved descriptors, perform a level j search in the second reconstructed data of the P 3D map descriptors to obtain Q 3D map descriptors.

[0348] The following explanation uses the example of the second reconstructed data being the k-th reconstructed data.

[0349] Based on partial or complete data of the retrieved descriptors, perform a level j search on the k-th reconstructed data of P 3D map descriptors to obtain Q 3D map descriptors.

[0350] S, P, Q, i, j, k, and m are positive integers, where 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrieval levels or the retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

[0351] For the specific explanation of S104C, reference can be made to Figure 5A the explanation of S104A in the illustrated embodiment, which will not be elaborated here.

[0352] Of course, it can be understood that after S104C, a conditional judgment step can be executed. When the condition is met, the retrieval of this embodiment can be ended. When the condition is not met, one or more levels of retrieval can be continued. This condition can be any condition, which can be reasonably set according to requirements. For example, the condition can be that the similarity or relevance is higher than or equal to a preset threshold, or the number of 3D map descriptors obtained by the retrieval is less than or equal to a preset number, etc. The embodiments of this application will not list them one by one.

[0353] The retrieval method of the 3D map in the embodiments of this application is a fragment process. Before and / or after this fragment process, other levels of retrieval can also be included. In other words, the retrieval method of the 3D map can include Figure 5C more levels of multi-level retrieval than those shown, such as 3 levels, 4 levels, or 5 levels, etc. For example, before S101C in the Figure 5C illustrated embodiment, there can be more levels of retrieval, or after S104C in the Figure 5C illustrated embodiment, there can be more levels of retrieval. The embodiments will not list them one by one here.

[0354] In some embodiments, the Q 3D map descriptors obtained through multi-level retrieval can be used for positioning. If the 3D map points corresponding to N of the Q 3D map descriptors match the 3D map points corresponding to the retrieval descriptor, the pose information of the electronic device can be obtained based on the 3D map points corresponding to the N 3D map descriptors. The electronic device can be the electronic device that collects the above visual information. 0 < N ≤ Q.

[0355] In this embodiment, the compressed data of S 3D map descriptors is decompressed at level m to obtain the first reconstructed data of S 3D map descriptors. Based on partial or complete data of the retrieved descriptors, a level i search is performed on the first reconstructed data of the S 3D map descriptors to filter out P 3D map descriptors. The compressed data of the P 3D map descriptors is decompressed at level k to obtain the second reconstructed data of P 3D map descriptors. Based on partial or complete data of the retrieved descriptors, a level j search is performed on the second reconstructed data of the P 3D map descriptors to filter out a smaller number of 3D map descriptors. The reconstructed data of the 3D map descriptors obtained through decompression at level one is alternated with the level one search. At least two levels of search involve staged decompression at level one, which can improve the search speed. The differentiated decompression settings of at least two levels of search can ensure the search accuracy. The degree of decompression or the degree of distortion of the reconstructed data of the 3D map descriptors used by any two levels of search are different. Compared to searching in the reconstructed data of a fully decompressed 3D map, this method can improve search speed and ensure search accuracy.

[0356] See Figure 5D , Figure 5D This is a schematic diagram of a fragment of a 3D map retrieval method provided in an embodiment of this application. In some implementations, this method can be applied to... Figures 1 to 4f Any of the example electronic devices, or those that can also be applied to Figures 1 to 4f Any of the example servers. The method includes, but is not limited to, the following steps:

[0357] S101D: Perform m-th level decompression on the compressed data of S 3D map descriptors to obtain the reconstructed data of S 3D map descriptors. The S 3D map descriptors correspond to multiple 3D map points in the 3D map.

[0358] The reconstructed data can refer to any one of the first reconstructed data, the second reconstructed data, ..., the m-th reconstructed data, where m is a positive integer. It should be understood that the terms "first," "second," and "third" in the reconstructed data series are not sequential; they are merely used to distinguish them. The following explanation uses the example of the m-th reconstructed data.

[0359] The compressed data of S 3D map descriptors is decompressed at the m-th level to obtain the first reconstructed data of S 3D map descriptors. The S 3D map descriptors correspond to multiple 3D map points in the 3D map.

[0360] S102D. Based on partial or complete data of the retrieved descriptors, perform the i-th level search in the reconstructed data of the S 3D map descriptors to obtain P 3D map descriptors.

[0361] Among them, the explanatory notes of S101D and S102D can be found in Figure 5C the explanatory notes of S101C and S102C, which will not be elaborated here.

[0362] S103D: Extract the binary data of P 3D map descriptors from the compressed data of P 3D map descriptors.

[0363] S104D: Perform the j-th level retrieval in the binary data of P 3D map descriptors according to the binary data of the retrieval descriptor to obtain Q 3D map descriptors.

[0364] S, P, Q, i, m, and j are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, where T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the total number of retrieval levels or the retrieval level threshold of the 3D map retrieval, and L is a positive integer greater than 1.

[0365] Among them, the explanatory notes of S103D and S104D can be found in Figure 5B the explanatory notes of S103B and S104B, which will not be elaborated here.

[0366] In this embodiment, the compressed data of S 3D map descriptors is decompressed at the m-th level to obtain the reconstructed data of S 3D map descriptors. According to part or all of the data of the retrieval descriptor, the i-th level retrieval is performed in the reconstructed data of S 3D map descriptors to screen out P 3D map descriptors. The binary data of the P 3D map descriptors is extracted from the compressed data of the P 3D map descriptors. According to the binary data of the retrieval descriptor, the j-th level retrieval is performed in the binary data of the P 3D map descriptors to screen out a smaller number of 3D map descriptors. The extraction or decompression from the compressed data and the retrieval are alternated to perform multi-level retrieval in the 3D map to obtain 3D map descriptors that can be used for positioning. There is one level of retrieval that is decompression in at least two levels of retrieval, which can improve the retrieval accuracy. There is one level of retrieval that is extraction from the compressed data in at least two levels of retrieval, which can improve the retrieval speed. Compared with retrieving in the reconstructed data of the 3D map obtained by complete decompression, the 3D map retrieval method provided by the embodiment of the present application can improve the retrieval speed and ensure the retrieval accuracy rate.

[0367] It should be noted that the above Figures 5A to 5C shown retrieval method processes can be flexibly combined with each other to form various different multi-level retrieval methods.

[0368] See Figure 6 , Figure 6This is a flowchart illustrating a 3D map retrieval method provided in an embodiment of this application. In some implementations, this method can be applied to... Figures 1 to 4f Any of the example electronic devices, or those that can also be applied to Figures 1 to 4f Any of the example servers. This embodiment uses a four-level multi-level search as an example for illustration. The first and second level searches use a search method based on a first distance, while the third and fourth level searches use a search method based on a second distance. The method includes, but is not limited to, the following steps:

[0369] The following method steps of the embodiments of this application shall be executed.

[0370] S201. Obtain the retrieval descriptor. This retrieval descriptor is extracted from visual information collected by the sensors of the electronic device, and the features corresponding to the real environment are extracted.

[0371] For an explanation of S201, please refer to [link / reference needed]. Figure 5A The explanation of S102A in the illustrated embodiment will not be repeated here.

[0372] S202. Extract the first (n=1) binarized data of the S1 3D map descriptors from the compressed data of the S1 3D map descriptors, and obtain the first (n=1) binarized data of the retrieved descriptor.

[0373] S1 3D map descriptors can be S1 region descriptors or S1 3D map point descriptors. One region descriptor corresponds to multiple 3D map points; for a detailed explanation, please refer to the explanations in the foregoing embodiments, which will not be repeated here.

[0374] For an explanation of the compressed data from the S1 3D map descriptors, please refer to [link / reference needed]. Figures 5A to 5D The embodiments shown are not described in detail here.

[0375] The first binarized data of the retrieval descriptor can be a partial binarized data of the retrieval descriptor. As an example, the executing entity in this embodiment can perform binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor, and select a portion from the binarized data of the retrieval descriptor as the first binarized data of the retrieval descriptor.

[0376] S203. Based on the first (n=1) binarized data of the search descriptors, perform the first (i=1) level search in the first binarized data of the S1 3D map descriptors to obtain P1 3D map descriptors.

[0377] The execution entity in this embodiment can use the first binarized data of the retrieval descriptor to perform a retrieval based on a first distance in the first (n=1) binarized data of S1 3D map descriptors, to obtain P1 3D map descriptors that are most similar to or most relevant to the first binarized data of the retrieval descriptor. P1 is less than S1. Alternatively, it can be understood that a subset is obtained through retrieval, and the number of 3D map descriptors in this subset is less than the number of 3D map descriptors in the original set of 3D map descriptors.

[0378] S204. Extract the second (n=2) binarized data of P1 3D map descriptors from the compressed data of P1 3D map descriptors, and obtain the second (n=2) binarized data of the retrieved descriptor.

[0379] The second (n=2) binarized data of the retrieval descriptor can be part or all of the binarized data of the retrieval descriptor. The number of bits in the second (n=2) binarized data of the retrieval descriptor differs from the number of bits in the first (n=1) binarized data of the retrieval descriptor. For example, the number of bits in the second (n=2) binarized data of the retrieval descriptor is greater than the number of bits in the first (n=1) binarized data of the retrieval descriptor. In this embodiment, the bit overhead of the second binarized data can be greater than the bit overhead of the first binarized data, which can improve the retrieval speed of the previous level retrieval and ensure the accuracy of the subsequent level retrieval.

[0380] S205. Based on the second (n=2) binarized data of the retrieved descriptors, perform a second (j=2) level search on the second (n=2) binarized data of P1 3D map descriptors to obtain Q1 3D map descriptors.

[0381] The execution entity in this embodiment can use the second binarized data of the retrieved descriptors to perform a search based on a first distance within the second binarized data of P1 3D map descriptors, to obtain Q1 3D map descriptors that are most similar to or most relevant to the second binarized data of the retrieved descriptors. Q1 is less than or equal to P1. Alternatively, it can be understood that another subset is obtained through the search, and the number of 3D map descriptors in this subset is less than the number of 3D map descriptors in the set of 3D map descriptors obtained in step 203.

[0382] S206. Perform the first (m=1) level decompression on the compressed data of Q1 3D map descriptors to obtain the first (m=1) reconstructed data of Q1 3D map descriptors.

[0383] S207. Based on partial or complete data of the retrieved descriptors, perform a third-level search in the first (m=1) reconstructed data of the Q1 3D map descriptors to obtain the Q2 3D map descriptors.

[0384] In one example, the executing entity of this embodiment can use partial or all of the data of the retrieved descriptors to perform a search based on a second distance in the first (m=1) reconstructed data of the Q1 descriptors to obtain Q2 3D map descriptors that are most similar to or most relevant to the partial or all of the retrieved descriptors. Q2 is less than or equal to Q1. Alternatively, it can be understood that another subset is obtained through retrieval, and the number of 3D map descriptors in this subset is less than the number of 3D map descriptors in the set of 3D map descriptors obtained in step 205.

[0385] Compared to the first two levels of retrieval based on the first distance, the third level uses retrieval based on the second distance, which can improve retrieval accuracy.

[0386] S208. Decompress the second (k=2) level of Q2 3D map descriptors to obtain the second (k=2) reconstructed data of Q2 3D map descriptors.

[0387] In one example, the distortion levels of the first (m=1) reconstructed data and the second (k=2) reconstructed data of the 3D map descriptor are different. In this embodiment, the distortion level of the first (m=1) reconstructed data of the 3D map descriptor is higher than that of the second (k=2) reconstructed data, which ensures the accuracy of subsequent level retrieval.

[0388] S209. Based on all the data of the retrieved descriptors, perform a fourth-level search in the second (k=2) reconstructed data of the Q2 3D map descriptors to obtain the Q3 3D map descriptors.

[0389] In one example, the executing entity of this embodiment can use all the data of the retrieved descriptors to search in the second (k=2) reconstructed data of Q2 3D map descriptors to obtain Q3 3D map descriptors that are most similar to or most relevant to all the data of the retrieved descriptors. Q3 is less than or equal to Q2. Alternatively, it can be understood that another subset is obtained through retrieval, and the number of 3D map descriptors in this subset is less than the number of 3D map descriptors in the set of 3D map descriptors obtained in step 207.

[0390] S210. Based on the 3D map points corresponding to the Q3 3D map descriptors, perform localization to obtain the pose information of the electronic device.

[0391] In this embodiment, by extracting binarized data of 3D map descriptors from compressed data or decompressing it to obtain reconstructed data of 3D map descriptors, and alternating this process with first-level retrieval, a multi-level retrieval can be performed on the 3D map to obtain 3D map descriptors that can be used for positioning. Compared to retrieving data from fully decompressed 3D map reconstructed data, this method can improve retrieval speed and ensure retrieval accuracy.

[0392] See Figure 7 , Figure 7 This is a flowchart illustrating a 3D map retrieval method provided in an embodiment of this application. In some implementations, this method can be applied to... Figures 1 to 4f Any of the example electronic devices, or those that can also be applied to Figures 1 to 4f Any of the example servers. This embodiment uses... Figure 5A The S 3D map descriptors in the illustrated embodiment are used as examples to illustrate S representative 3D map descriptors. Each of the S representative 3D map descriptors corresponds to at least one data set. The above... Figure 5A The specific methods of 102A may include, but are not limited to, the following steps:

[0393] S1021. Based on the binarized data of the retrieved descriptors, perform the i-th level retrieval in the binarized data of the S representative 3D map descriptors to obtain at least one representative 3D map descriptor.

[0394] At least one representative 3D map descriptor corresponds to at least one dataset, and each dataset contains compressed data of multiple 3D map descriptors. The 3D map descriptors in each dataset have a certain degree of correlation or similarity.

[0395] S binary data representing 3D map descriptors are used. The binary data for retrieving a descriptor can be part or all of the binary data of the retrieved descriptor. The execution entity in this embodiment can use the binary data of the retrieved descriptor to perform a search based on a first distance among the S binary data representing 3D map descriptors to obtain at least one representative 3D map descriptor that is most similar to or most relevant to the binary data of the retrieved descriptor. The number of at least one representative 3D map descriptor is less than or equal to S.

[0396] S1022. Take at least one 3D map descriptor from the data set corresponding to each 3D map descriptor as P 3D map descriptors.

[0397] By retrieving at least one representative 3D map descriptor, at least one data set is obtained, thereby obtaining compressed data of P 3D map descriptors in the at least one data set.

[0398] For example, S representative 3D map descriptors are T1, T2, and T3, where T1 corresponds to data set 1, T2 corresponds to data set 2, and T3 corresponds to data set 3. By retrieving at least one representative 3D map descriptor, T3, data set 3 is obtained, which may include compressed data of P 3D map descriptors.

[0399] In this embodiment, a first-level search is performed using a smaller number of representative 3D map descriptors to improve search speed. Subsequently, first-level or multi-level searches can be performed on the datasets corresponding to these representative 3D map descriptors to improve search accuracy. Compared to searching through the reconstructed 3D map data obtained after complete decompression, this method improves search speed while ensuring search accuracy.

[0400] It should be noted that Figures 5C to 5D The S 3D map descriptors in the illustrated embodiment are S specific implementations representing 3D map descriptors, and are consistent with the above. Figure 7 The illustrated embodiment is similar. The above... Figure 5B 102B, or Figure 5C 102C, or Figure 5D For details on the 102D, please refer to [the relevant documentation / reference]. Figure 7 The embodiments shown are not described in detail here.

[0401] See Figure 8 and Figure 9 , Figure 8 This is a flowchart illustrating a 3D map retrieval method provided in an embodiment of this application. Figure 9 This is a schematic diagram illustrating the processing steps of a 3D map retrieval method provided in this application embodiment. This embodiment includes a server and an electronic device. The method includes, but is not limited to, the following steps:

[0402] S401. The server determines the magnitude relationship between each component of the S 3D map descriptors and the corresponding component of the preset threshold vector based on each component of the S 3D map descriptors and the corresponding component of the preset threshold vector.

[0403] S402. The server performs binarization processing on the above size relationship to obtain the binarized data of each of the S 3D map descriptors.

[0404] S403. The server performs quantization processing on the absolute value of the difference between each component of the S 3D map descriptors and the corresponding component of the preset threshold vector, and obtains the quantized data of each of the S 3D map descriptors.

[0405] The components included in the preset threshold vector are arbitrary values.

[0406] S404. The server encapsulates the binarized and quantized data of each of the S 3D map descriptors to obtain the 3D map bitstream.

[0407] S405, The server sends the 3D map bitstream to the electronic device.

[0408] S406. Electronic devices obtain search descriptors.

[0409] Reference Figure 9 As shown, electronic devices can extract retrieval descriptors from visual information collected by sensors.

[0410] S407. The electronic device decapsulates the bitstream of the 3D map to obtain the binarized data and quantized data of each of the S 3D map descriptors, and extracts the binarized data of each of the S 3D map descriptors from them.

[0411] Reference Figure 9 As shown, taking five 3D map descriptors (T1, T2, T3, T4, and T5) as an example, S 3D map descriptors can be decapsulated by an electronic device to obtain the following: Figure 9 The five 3D map descriptors shown are represented by their respective binarized and quantized data.

[0412] S408. The electronic device performs a first-level search on the binary data of each of the S 3D map descriptors based on the binary data of the search descriptors, so as to obtain P 3D map descriptors.

[0413] The binarized data of the retrieval descriptor is obtained as follows: The electronic device can determine the magnitude relationship between each component of the retrieval descriptor and the corresponding component of the preset threshold vector based on each component of the retrieval descriptor and the corresponding component of the preset threshold vector. This magnitude relationship is then binarized to obtain the binarized data of the retrieval descriptor. The first-level retrieval in the binarized data of the S 3D map descriptors can be performed using a retrieval method based on a first distance.

[0414] by Figure 9 To further illustrate, based on the binarized data of the search descriptors, a first-level search is performed among five 3D map descriptors (T1, T2, T3, T4, and T5). For example, the Hamming distance between the binarized data of each of the five 3D map descriptors (T1, T2, T3, T4, and T5) and the binarized data of the search descriptor can be calculated, and the 3D map descriptors can be filtered based on the Hamming distance. Here, we take filtering out two 3D map descriptors as an example, that is, filtering out... Figure 9 The dashed boxes indicate T1 and T2. That is, P = 2.

[0415] S409. The electronic device decompresses the quantized data of each of the P 3D map descriptors to obtain the reconstructed data of each of the P 3D map descriptors.

[0416] An electronic device can perform inverse quantization processing on the quantized data of each of the P 3D map descriptors to obtain the inverse quantized data of each of the P 3D map descriptors. Based on the inverse quantized data and binarized data of each of the P 3D map descriptors, the reconstructed data of each of the P 3D map descriptors is obtained.

[0417] S410. The electronic device performs a second-level search on the reconstructed data of each of the P 3D map descriptors based on the search descriptors, so as to obtain Q 3D map descriptors.

[0418] Second-level retrieval can be performed on the reconstructed data of each of the P 3D map descriptors using a retrieval method based on a second distance.

[0419] by Figure 9 To further illustrate, a second-level search is performed in T1 and T2 based on all components of the retrieved descriptor. For example, the Euclidean distance between the reconstructed data and the retrieved descriptor in T1 and T2 can be calculated separately, and 3D map descriptors can be selected based on the Euclidean distance. Here, we take selecting one 3D map descriptor as an example, that is, selecting... Figure 9 The value shown is T1, which means Q = 1.

[0420] In this embodiment, the server compresses S 3D map descriptors to obtain binarized and quantized data for each of the S 3D map descriptors, reducing the resource overhead required for transmitting the 3D map. The binarized data of the S 3D map descriptors is acquired via an electronic device. Based on the binarized data of the retrieved descriptor and the S 3D map descriptors, a retrieval method based on a first distance is used to quickly filter and obtain P 3D map descriptors, thus improving retrieval speed. The reconstructed data of the P 3D map descriptors is acquired via an electronic device. Based on the retrieved descriptor and the reconstructed data of the P 3D map descriptors, a retrieval method based on a second distance is used to accurately filter and obtain Q 3D map descriptors, thus improving retrieval accuracy.

[0421] The 3D map retrieval method of this application embodiment has been described in detail above with reference to the accompanying drawings. The following section will further describe the method in conjunction with... Figure 10 and Figure 11 This application describes a 3D map retrieval apparatus according to embodiments of the present application. It should be understood that the 3D map retrieval apparatus is capable of executing the 3D map retrieval method according to embodiments of the present application. To avoid unnecessary repetition, repeated descriptions are appropriately omitted below when describing the 3D map retrieval apparatus according to embodiments of the present application.

[0422] See Figure 10 , Figure 10 This is a schematic diagram of the structure of a 3D map retrieval device provided in an embodiment of this application. Figure 10 As shown, the 3D map retrieval device 1000 may include: a retrieval module 1001, an extraction module 1002, and a decompression module 1003.

[0423] In a first possible implementation manner, an extraction module 1002 is configured to extract binarized data of S 3D map descriptors from compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in a 3D map; a retrieval module 1001 is configured to perform an i-level retrieval on the binarized data of the S 3D map descriptors according to the binarized data of a retrieval descriptor, so as to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to a real environment extracted from visual information collected by a sensor of an electronic device; a decompression module 1003 is configured to perform an m-level decompression on the compressed data of the P 3D map descriptors to obtain reconstructed data of the P 3D map descriptors; the decompression process of the compressed data of the P 3D map descriptors at least includes the m-level decompression; the retrieval module 1001 is configured to perform a j-level retrieval on the reconstructed data of the P 3D map descriptors according to partial data or all data of the retrieval descriptor, so as to obtain Q 3D map descriptors, where S, P, Q, i, j, and m are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrieval levels or a retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

[0424] In some embodiments, N of the Q 3D map descriptors are used for positioning, and 3D map points corresponding to the N 3D map descriptors match 3D map points corresponding to the retrieval descriptor, where N is a positive integer and 0 < N ≤ Q. The apparatus further includes: a pose determination module; the pose determination module is configured to perform positioning according to the 3D map points corresponding to the N 3D map descriptors, so as to obtain pose information of the electronic device.

[0425] In some embodiments, the retrieval method adopted for the i-level retrieval is a retrieval method based on a first distance, and the retrieval method adopted for the j-level retrieval is a retrieval method based on a second distance.

[0426] In some embodiments, the apparatus further includes: an acquisition module, configured to: receive the retrieval descriptor and perform binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor; or, receive the visual information, extract the retrieval descriptor from the visual information, and perform binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor; or, in response to an operation of collecting visual information input by a user, trigger the sensor to collect visual information of the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and perform binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor.

[0427] In some embodiments, when N < Q, the decompression module 1003 is further configured to: perform k-level decompression on the compressed data of the Q 3D map descriptors to obtain the reconstructed data of the Q 3D map descriptors; perform r-level retrieval on the reconstructed data of the Q 3D map descriptors according to part or all of the data of the retrieval descriptor to obtain the N 3D map descriptors; where N < Q < P, the decompression process of the compressed data of the P 3D map descriptors includes the m-level decompression and the k-level decompression, r and k are positive integers, m < k, j < r ≤ L.

[0428] In some embodiments, the compressed data of the P 3D map descriptors includes the binarized data and the quantization data of each of the P 3D map descriptors, and the decompression module 1003 is specifically configured to: perform inverse quantization processing on the quantization data of each of the P 3D map descriptors to obtain P inverse quantization data, and the P inverse quantization data is used as the reconstructed data of the P 3D map descriptors; perform inverse quantization processing on the quantization data of each of the Q 3D map descriptors to obtain Q inverse quantization data; and obtain the reconstructed data of each of the Q 3D map descriptors according to the Q inverse quantization data and the binarized data of each of the Q 3D map descriptors.

[0429] In some embodiments, when N = Q, the compressed data of the P 3D map descriptors includes the binarized data and the quantization data of each of the P 3D map descriptors, and the decompression module 1003 is specifically configured to: perform inverse quantization processing on the quantization data of each of the P 3D map descriptors to obtain P inverse quantization data; and obtain the reconstructed data of each of the P 3D map descriptors according to the P inverse quantization data and the binarized data of each of the P 3D map descriptors.

[0430] In some embodiments, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors respectively corresponds to at least one data set, and each of the at least one data sets includes at least one 3D map descriptor. The retrieval module 1001 is configured to: perform i-level retrieval on the binarized data of the S representative 3D map descriptors according to the binarized data of the retrieval descriptor to obtain at least one representative 3D map descriptor; and use the 3D map descriptors in the data sets corresponding to the at least one representative 3D map descriptor as the P 3D map descriptors.

[0431] In a second possible implementation, an extraction module 1002 is configured to extract first binary data of the S 3D map descriptors from compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in a 3D map; a retrieval module 1001 is configured to perform an i-level retrieval in the first binary data of the S 3D map descriptors according to the first binary data of a retrieval descriptor to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to a real environment extracted from visual information collected by a sensor of an electronic device; the extraction module 1002 is further configured to extract second binary data of the P 3D map descriptors from compressed data of the P 3D map descriptors; the retrieval module 1001 is configured to perform a j-level retrieval in the second binary data of the P 3D map descriptors according to the second binary data of the retrieval descriptor to obtain Q 3D map descriptors, where S, P, Q, i, and j are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrieval levels or a retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

[0432] In some embodiments, N of the Q 3D map descriptors are used for positioning, and 3D map points corresponding to the N 3D map descriptors match 3D map points corresponding to the retrieval descriptor, where N is a positive integer and 0 < N ≤ Q. The apparatus further includes: a pose determination module; the pose determination module is configured to perform positioning according to the 3D map points corresponding to the N 3D map descriptors to obtain pose information of the electronic device.

[0433] In some embodiments, the retrieval methods used for the i-level retrieval and the j-level retrieval are both retrieval based on a first distance. The P 3D map descriptors belong to the S 3D map descriptors, and positions of the first binary data of each of the P 3D map descriptors in the compressed data of the 3D map descriptors are different from positions of the second binary data of the 3D map descriptors in the compressed data of the 3D map descriptors, where P < S.

[0434] In some embodiments, the length of the first binary data of each of the P 3D map descriptors is less than the length of the second binary data of each of the P 3D map descriptors.

[0435] In some embodiments, the apparatus further includes: an acquisition module, configured to: receive the retrieval descriptor and binarize the retrieval descriptor to obtain first binarized data and second binarized data of the retrieval descriptor; or, receive the visual information, extract the retrieval descriptor from the visual information, and binarize the retrieval descriptor to obtain first binarized data and second binarized data of the retrieval descriptor; or, in response to a user-input visual information acquisition operation, trigger the sensor to acquire visual information from the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and binarize the retrieval descriptor to obtain first binarized data and second binarized data of the retrieval descriptor.

[0436] In some embodiments, the length of the first binarized data of the retrieval descriptor is equal to the length of the first binarized data of each of the S 3D map descriptors, and / or, the length of the second binarized data of the retrieval descriptor is equal to the length of the second binarized data of each of the S 3D map descriptors.

[0437] In some embodiments, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. The retrieval module 1001 is specifically configured to: perform an i-th level retrieval in the first binarized data of the S representative 3D map descriptors based on the first binarized data of the retrieval descriptor to obtain at least one representative 3D map descriptor; and use the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0438] In a third possible implementation, a decompression module 1003 is configured to perform m-level decompression on the compressed data of S 3D map descriptors to obtain first reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in a 3D map; a retrieval module 1001 is configured to perform i-level retrieval in the first reconstructed data of the S 3D map descriptors according to partial or all of the retrieval descriptor data to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to a real environment extracted from visual information collected by a sensor of an electronic device; the decompression module 1003 is further configured to perform k-level decompression on the compressed data of the P 3D map descriptors to obtain second reconstructed data of the P 3D map descriptors, the decompression process of the compressed data of the S 3D map descriptors includes the m-level decompression and the k-level decompression, or the decompression process of the compressed data of the S 3D map descriptors includes the k-level decompression, and the k-level decompression includes the m-level decompression; the retrieval module 1001 is configured to perform j-level retrieval in the second reconstructed data of the P 3D map descriptors according to partial or all of the retrieval descriptor to obtain Q 3D map descriptors, where S, P, Q, i, j, k, and m are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrieval levels or the retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

[0439] In some embodiments, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor, where N is a positive integer and 0 < N ≤ Q. The apparatus further includes: a pose determination module; the pose determination module is configured to perform positioning according to the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0440] In some embodiments, the retrieval methods used for the i-th level retrieval and the j-th level retrieval are both based on the second distance retrieval method. The P 3D map descriptors belong to the S 3D map descriptors. The distortion degree of the first reconstructed data of each of the P 3D map descriptors is different from the distortion degree of the second reconstructed data of each of the P 3D map descriptors. The distortion degree of the first reconstructed data of each of the P 3D map descriptors is used to represent the degree of difference between the first reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor. The distortion degree of the second reconstructed data of each of the P 3D map descriptors is used to represent the degree of difference between the second reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor. Wherein, P < S.

[0441] In some embodiments, the distortion of the first reconstructed data of each of the P 3D map descriptors is greater than the distortion of the second reconstructed data of each of the P 3D map descriptors.

[0442] In some embodiments, the apparatus further includes: an acquisition module, configured to: receive the retrieval descriptor and acquire partial or all data of the retrieval descriptor; or, receive the visual information, extract the retrieval descriptor from the visual information, and acquire partial or all data of the retrieval descriptor; or, in response to a user-input visual information acquisition operation, trigger the sensor to acquire visual information of the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and acquire partial or all data of the retrieval descriptor.

[0443] In some embodiments, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. The retrieval module 1001 is specifically used to: perform a level i retrieval on the m-th reconstructed data of the S representative 3D map descriptors based on partial or all data of the retrieval descriptor to obtain at least one representative 3D map descriptor; and use the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0444] In a fourth possible implementation, a decompression module 1003 is configured to perform m-level decompression on compressed data of S 3D map descriptors to obtain reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in a 3D map; a retrieval module 1001 is configured to perform i-level retrieval in the reconstructed data of the S 3D map descriptors according to partial data or all data of a retrieval descriptor, to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to a real environment extracted from visual information collected by a sensor of an electronic device; an extraction module 1002 is configured to extract binarized data of the P 3D map descriptors from the compressed data of the P 3D map descriptors; the retrieval module 1001 is further configured to perform j-level retrieval in the binarized data of the P 3D map descriptors according to the binarized data of the retrieval descriptor, to obtain Q 3D map descriptors, where S, P, Q, i, m, and j are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrieval levels or a retrieval level threshold of the 3D map retrieval, and L is a positive integer greater than 1.

[0445] In some embodiments, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor, where N is a positive integer and 0 < N ≤ Q. The apparatus further includes: a pose determination module; the pose determination module is configured to perform positioning according to the 3D map points corresponding to the N 3D map descriptors, to obtain pose information of the electronic device.

[0446] In some embodiments, the retrieval method adopted for the i-level retrieval is a retrieval method based on a second distance, and the retrieval method adopted for the j-level retrieval is a retrieval method based on a first distance.

[0447] In some embodiments, the apparatus further includes: an acquisition module, configured to: receive the retrieval descriptor and perform binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor; or, receive the visual information, extract the retrieval descriptor from the visual information, and perform binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor; or, in response to an operation of collecting visual information input by a user, trigger the sensor to collect visual information of the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and perform binarization processing on the retrieval descriptor to obtain binarized data of the retrieval descriptor.

[0448] In some embodiments, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one data set, and each of the at least one data set includes at least one 3D map descriptor. The retrieval module 1001 is specifically used to: perform a level i retrieval on the m-th reconstructed data of the S representative 3D map descriptors based on partial or all data of the retrieval descriptor to obtain at least one representative 3D map descriptor; and use the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0449] It should be noted that the 3D map retrieval device 1000 can perform... Figures 5A to 5D any or Figure 6 or Figure 7 The 3D map retrieval device method of the illustrated embodiment, or, executing Figure 8 The relevant content of S406-S410 in the illustrated embodiment, or, execution Figure 9 The relevant content of the illustrated embodiment is as follows. For specific implementation principles and technical effects, please refer to the detailed explanation of the above method embodiments; they will not be repeated here.

[0450] Figure 11 This is a schematic block diagram illustrating one implementation of the decoding apparatus 1100 used in embodiments of this application. The decoding apparatus 1100 may include a processor 1101, a memory 1102, and a bus system 1103. The processor 1101 and the memory 1102 are connected via the bus system 1103. The memory 1102 stores instructions, and the processor 1101 executes the instructions stored in the memory 1102 to perform various 3D map retrieval methods described in this application. To avoid repetition, further details are omitted here.

[0451] In this embodiment, the processor 1101 may be a central processing unit (CPU), or it may be other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0452] The memory 1102 may include a ROM device or a RAM device. Any other suitable type of storage device may also be used as memory 1102. Memory 1102 may include code and data 11021 accessed by processor 1101 using bus 1103. Memory 1102 may further include an operating system 11023 and an application program 11022, which includes at least one program that allows processor 1101 to execute the 3D map retrieval method described in this application. For example, application program 11022 may include applications 1 to N, which further include a 3D map application that executes the 3D map retrieval method described in this application.

[0453] In addition to the data bus, the bus system 1103 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 1103 in the figure.

[0454] Optionally, the decoding device 1100 may also include one or more output devices, such as a display 1104. In one example, the display 1104 may be a haptic display that combines a display with a haptic unit capable of operatively sensing touch input. The display 1104 may be connected to the processor 1101 via a bus 1103.

[0455] It should be noted that the decoding device 1100 can execute the 3D map retrieval method in this application.

[0456] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.

[0457] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically addressed to non-temporary tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0458] Instructions can be executed by one or more processors, such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the various illustrative logic blocks, modules, and steps described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.

[0459] The technology of this application can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described in this application are intended to emphasize functional aspects of the apparatus for performing the disclosed technology, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units can be combined with suitable software and / or firmware within a codec hardware unit, or provided via interoperable hardware units (containing one or more processors as described above).

[0460] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for retrieving 3D maps, characterized in that, The method includes: Extracting the binarized data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points in the 3D map; Performing an i-level retrieval in the binarized data of the S 3D map descriptors according to the binarized data of the retrieval descriptor to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to the real environment extracted from the visual information collected by the sensor of the electronic device; Performing an m-level decompression on the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors; the decompression process of the compressed data of the P 3D map descriptors includes at least the m-level decompression; Performing a j-level retrieval in the reconstructed data of the P 3D map descriptors according to part or all of the data of the retrieval descriptor to obtain Q 3D map descriptors, where S, P, Q, i, j, and m are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of retrievals or the retrieval level threshold of the 3D map, and L is a positive integer greater than 1.

2. The method according to claim 1, characterized in that, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor, where N is a positive integer, 0 < N ≤ Q, and the method further includes: Performing positioning according to the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

3. The method according to claim 1 or 2, characterized in that, The retrieval method used in the i-level retrieval is a retrieval method based on the first distance, and the retrieval method used in the j-level retrieval is a retrieval method based on the second distance.

4. The method according to claim 1 or 2, characterized in that, The method further includes: Receiving the retrieval descriptor and performing binarization processing on the retrieval descriptor to obtain the binarized data of the retrieval descriptor; or, Receiving the visual information, extracting the retrieval descriptor from the visual information, and performing binarization processing on the retrieval descriptor to obtain the binarized data of the retrieval descriptor; or, Responding to the operation of visual information collection input by the user, triggering the sensor to collect visual information on the real environment, obtaining the visual information, extracting the retrieval descriptor from the visual information, and performing binarization processing on the retrieval descriptor to obtain the binarized data of the retrieval descriptor.

5. The method according to claim 2, characterized in that, In the case of N < Q, the method further includes: Performing a k-level decompression on the compressed data of the Q 3D map descriptors to obtain the reconstructed data of the Q 3D map descriptors; Performing an r-level retrieval in the reconstructed data of the Q 3D map descriptors according to part or all of the data of the retrieval descriptor to obtain the N 3D map descriptors; Among them, N < Q < P. The decompression process of the compressed data of the P 3D map descriptors includes the m-th level decompression and the k-th level decompression, where r and k are positive integers, m < k, and j < r ≤ L.

6. The method according to claim 5, characterized in that, The compressed data of the P 3D map descriptors includes the binarized data and the quantized data of each of the P 3D map descriptors. The m-th level decompression of the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors includes: Performing inverse quantization on the quantized data of each of the P 3D map descriptors to obtain P inverse quantized data, and using the P inverse quantized data as the reconstructed data of the P 3D map descriptors; The k-th level decompression of the compressed data of the Q 3D map descriptors to obtain the reconstructed data of the Q 3D map descriptors includes: Performing inverse quantization on the quantized data of each of the Q 3D map descriptors to obtain Q inverse quantized data; According to the Q inverse quantized data and the binarized data of each of the Q 3D map descriptors, obtaining the reconstructed data of each of the Q 3D map descriptors.

7. The method according to claim 2, characterized in that, In the case where N = Q, the compressed data of the P 3D map descriptors includes the binarized data and the quantized data of each of the P 3D map descriptors. The m-th level decompression of the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors includes: Performing inverse quantization on the quantized data of each of the P 3D map descriptors to obtain P inverse quantized data; According to the P inverse quantized data and the binarized data of each of the P 3D map descriptors, obtaining the reconstructed data of each of the P 3D map descriptors.

8. The method according to any one of claims 1 to 2, 5 to 7, characterized in that, The S 3D map descriptors are S representative 3D map descriptors. Each of the S representative 3D map descriptors corresponds to at least one data set, and each data set of the at least one data set includes at least one 3D map descriptor. The i-th level retrieval in the binarized data of the S 3D map descriptors according to the binarized data of the retrieval descriptor to obtain P 3D map descriptors includes: Performing the i-th level retrieval in the binarized data of the S representative 3D map descriptors according to the binarized data of the retrieval descriptor to obtain at least one representative 3D map descriptor; Taking the 3D map descriptors in the data sets corresponding to each of the at least one representative 3D map descriptors as the P 3D map descriptors.

9. A 3D map retrieval device, characterized in that, The device includes: An extraction module, configured to extract the binarized data of the S 3D map descriptors from the compressed data of the S 3D map descriptors, and the S 3D map descriptors correspond to multiple 3D map points in the 3D map; A retrieval module, configured to perform the i-th level retrieval in the binarized data of the S 3D map descriptors according to the binarized data of the retrieval descriptor to obtain P 3D map descriptors; the retrieval descriptor is a feature corresponding to the real environment extracted from the visual information collected by the sensor of the electronic device. A decompression module, configured to perform m - level decompression on the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors; the decompression process of the compressed data of the P 3D map descriptors at least includes the m - level decompression; The retrieval module is configured to perform j - level retrieval in the reconstructed data of the P 3D map descriptors according to partial or all of the data of the retrieval descriptor, so as to obtain Q 3D map descriptors. S, P, Q, i, j, and m are positive integers, and 0 < Q < P, 0 < P < T, 0 < S ≤ T. T represents the total number of 3D map descriptors in the 3D map. j = i + 1, 1 ≤ i < L, 1 < j ≤ L. L represents the total number of retrieval levels or the retrieval - level threshold of the 3D map retrieval, and L is a positive integer greater than 1.

10. The apparatus according to claim 9, characterized in that, N of the Q 3D map descriptors are used for positioning. The 3D map points corresponding to the N 3D map descriptors match the 3D map points corresponding to the retrieval descriptor. N is a positive integer, 0 < N ≤ Q. The apparatus further includes: a pose determination module; The pose determination module is configured to perform positioning according to the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

11. The apparatus according to claim 9 or 10, characterized in that, The retrieval method adopted for the i - level retrieval is a retrieval method based on the first distance, and the retrieval method adopted for the j - level retrieval is a retrieval method based on the second distance.

12. The apparatus according to claim 9 or 10, characterized in that, The apparatus further includes: an acquisition module, configured to: Receive the retrieval descriptor and perform binarization processing on the retrieval descriptor to obtain the binarized data of the retrieval descriptor; or, Receive the visual information, extract the retrieval descriptor from the visual information, and perform binarization processing on the retrieval descriptor to obtain the binarized data of the retrieval descriptor; or, In response to an operation of collecting visual information input by the user, trigger the sensor to collect visual information about the real environment, obtain the visual information, extract the retrieval descriptor from the visual information, and perform binarization processing on the retrieval descriptor to obtain the binarized data of the retrieval descriptor.

13. The apparatus according to claim 10, characterized in that, In the case of N < Q, the decompression module is further configured to: Perform k - level decompression on the compressed data of the Q 3D map descriptors to obtain the reconstructed data of the Q 3D map descriptors; Perform r - level retrieval in the reconstructed data of the Q 3D map descriptors according to partial or all of the data of the retrieval descriptor to obtain the N 3D map descriptors; Wherein, N < Q < P. The decompression process of the compressed data of the P 3D map descriptors includes the m - level decompression and the k - level decompression. r and k are positive integers, m < k, j < r ≤ L.

14. The apparatus according to claim 13, characterized in that, The compressed data of the P 3D map descriptors includes the binarized data and quantization data of each of the P 3D map descriptors. The decompression module is specifically configured to: The quantized data of each of the P 3D map descriptors are dequantized to obtain P dequantized data, which are used as the reconstruction data of the P 3D map descriptors. The quantized data of each of the Q 3D map descriptors are dequantized to obtain Q dequantized data. Based on the Q inverse quantized data and the binarized data of the Q 3D map descriptors, the reconstructed data of the Q 3D map descriptors are obtained.

15. The apparatus according to claim 10, characterized in that, When N=Q, the compressed data of the P 3D map descriptors includes the binarized data and quantized data of each of the P 3D map descriptors, and the decompression module is specifically used for: The quantized data of each of the P 3D map descriptors are dequantized to obtain P dequantized data. Based on the P inverse quantized data and the binarized data of each of the P 3D map descriptors, the reconstructed data of each of the P 3D map descriptors is obtained.

16. The apparatus according to any one of claims 9 to 10, 13 to 15, characterized in that, The S 3D map descriptors are S representative 3D map descriptors, each of which corresponds to at least one data set. Each of the at least one data set includes at least one 3D map descriptor. The retrieval module is used for: Based on the binarized data of the retrieval descriptor, the i-th level retrieval is performed in the binarized data of the S representative 3D map descriptors to obtain at least one representative 3D map descriptor; The 3D map descriptors in the data set corresponding to each of the at least one 3D map descriptor are taken as the P 3D map descriptors.

17. A 3D map retrieval device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1-8.

Images