Method, device and computer program for searching 3D maps

Abstract

In order to help improve the retrieval performance, a method and an apparatus for retrieving a 3D map are provided, which belong to the field of 3D map technology. The method for retrieving a 3D map includes: obtaining binary data of S 3D map descriptors; based on the binary data of the retrieval descriptors (S101A), performing a stage i retrieval on the binary data of the S 3D map descriptors to obtain P 3D map descriptors through screening (S102A); performing a stage m decompression on the compressed data of the P 3D map descriptors to obtain reconstructed data of the P 3D map descriptors (S103A); and based on the partial data or the whole data of the retrieval descriptors, performing a stage j retrieval on the reconstructed data of the P 3D map descriptors to obtain Q 3D map descriptors (S104A). This can help improve the retrieval speed and ensure the retrieval accuracy.

Description

[Technical Field]

[0001] This application relates to positioning technology, and in particular to 3D maps. search for The present invention relates to a method and apparatus for [Background technology]

[0002] Virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies are emerging multimedia virtual scenario technologies in recent years. These technologies can be used to generate virtual reality and overlay it on the real world to create new visual environments and interactive experiences. In such applications, electronic devices need to determine their pose information in the current environment to accurately implement the blending between virtual objects and real scenes.

[0003] Furthermore, in applications such as autonomous driving, autonomous navigation, unmanned aerial vehicle automatic inspection, and industrial robotics, in order to perform accurate route planning, navigation, detection, and control, a transport device such as a vehicle, unmanned aerial vehicle, or robot needs to determine the pose of an electronic device being transported by the transport device, thereby determining the pose of the transport device in the current environment.

[0004] In the aforementioned application, for the problem that the pose of the electronic device in the current environment needs to be determined, a typical solution is as follows: the electronic device receives a three-dimensional (3D) map of the environment where the electronic device is located from a server or another device, collects visual information in the environment by using a local sensor, and based on the collected visual information, calculates a pose in the downloaded 3D map. search to determine the current pose of the electronic device.

[0005] However, the original 3D map usually contains a large data volume, search and the computational complexity for it is usually very high. This requires a large number of computing resources and takes a long time, thereby affecting the user experience.

Summary of the Invention

[0006] Embodiments of the present application provide a method and an apparatus for search for compressing a 3D map to improve performance. search

[0007] According to a first aspect, embodiments of the present application provide a method for search for compressing a 3D map. The method includes: extracting binary data of S 3D map descriptors from compressed data of S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map; search performing stage i in the binary data of the S 3D map descriptors based on the binary data of the descriptors to obtain P 3D map descriptors, where search the descriptors correspond to the real environment and are features extracted from visual information collected by a sensor of an electronic device; performing unfolding of stage m on the compressed data of the P 3D map descriptors to obtain reconstructed data of the P 3D map descriptors, where the unfolding process (complete unfolding process) of the compressed data of the P 3D map descriptors includes at least the unfolding of stage m; and search performing stage j in the reconstructed data of the P 3D map descriptors based on partial data or all data of the descriptors to obtain Q 3D map descriptors, where S, P, Q, i, j, and m are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, L represents the total number of search stages of the 3D map or search a stage number threshold, and L is a positive integer greater than 1. search search ​​​

[0008] In this embodiment, search Based on the binary data of the descriptor, in the binary data of S 3D map descriptors, at stage i search is executed to obtain P 3D map descriptors through screening, and the expansion at stage m is executed on the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors. search Based on the partial data or all data of the descriptor, it can be seen that at stage j in the reconstructed data of the P 3D map descriptors search is executed to obtain fewer 3D map descriptors through screening. The binary data of the compressed data is search used in one stage in at least two stages of search to search improve the speed, and the reconstructed data obtained through expansion is used in search of another stage to search ensure the accuracy. Compared with search in the reconstructed data of the 3D map obtained through full expansion, the method provided in this embodiment of the present application for search for the 3D map can search improve the performance. In other words, the method provided in this embodiment of the present application for search for the 3D map can search improve the performance.

[0009] Stage j can be an intermediate stage in the multi-stage search , that is, j < L, or the final stage in the multi-stage search , that is, j = L.

[0010] In an example, the expansion at stage m can be a stage-based expansion, can be understood as a sub-process of full expansion, or can be understood as partial expansion. In another example, the expansion at stage m can be full expansion.

[0011] In a possible design, N out of Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors, N is a positive integer, and 0 < N ≤ Q. The method may further include the step of performing positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain pose information of the electronic device.

[0012] In this embodiment, search when the 3D map points corresponding to the N 3D map descriptors obtained through search coincide with the map points corresponding to the descriptors, positioning can be performed based on the 3D map points corresponding to the N 3D map descriptors. This helps to improve the positioning performance.

[0013] In a possible design, the search used for search stage i is a search method based on the first distance, and the search used for search stage j is a search method based on the second distance.

[0014] The first distance may include a distance obtained by using binary data such as the Hamming distance. For example, the first distance may be the Hamming distance. In another example, the first distance may be a distance obtained by obtaining the absolute value of the difference value between corresponding bits of two binary data and summing up all the absolute values of the difference values. The search based on the first distance means determining the correlation or similarity by calculating the first distance and screening the 3D map descriptors. The second distance may include, but is not limited to, the Euclidean distance, the inner product distance, the cosine distance, the Manhattan distance, and the like. The search based on the second distance means determining the correlation or similarity by calculating the second distance and screening the 3D map descriptors.

[0015] In this embodiment, based on the first distance search teeth, search Speed may be improved based on the second distance search teeth, search The accuracy may be improved.

[0016] In a possible design, the method further comprises: search receiving a descriptor; and search Perform binarization on the descriptor search The method may further comprise obtaining binary data of the descriptor. search The executing entity (which may be an electronic device or may be a server (e.g., a cloud server)) configured to execute the method receives the data transmitted by another device. search For example, in the first aspect, search The subject of the method is configured to perform the method. ... search Receive the descriptor. Then, search Perform binarization on the descriptors, search Gets the binary data of the descriptor.

[0017] In one possible design, the method further comprises receiving visual information; search extracting descriptors from the visual information; and search Perform binarization on the descriptor search The method may further comprise obtaining binary data of the descriptor. search An executing entity (which may be an electronic device or a server) configured to execute the method receives visual information transmitted by another device, search Descriptors can then be extracted from the visual information. search Perform binarization on the descriptors, search Gets the binary data of the descriptor.

[0018] In a possible design, the method further comprises the steps of: triggering the sensor to perform visual information gathering on the real environment in response to a visual information gathering operation input by a user to acquire visual information; search The step of extracting descriptors from visual information, and search performing binarization on the descriptors to search obtain the binary data of the descriptors may be included. In the first aspect, search an execution entity (which may be an electronic device or a server) configured to execute the method in the present embodiment collects visual information and search extracts descriptors from the visual information. Next, search performs binarization on the descriptors to search obtain the binary data of the descriptors.

[0019] According to the method for generating a 3D map in the present embodiment of the present application, search for it can be executed in a 3D map based on descriptors in different application scenarios, which is widely applicable. search is based on descriptors in different application scenarios search and can be executed in a 3D map, which is widely applicable.

[0020] In a possible design, when N < Q, the method further includes: performing the expansion of stage k on the compressed data of Q 3D map descriptors to obtain the reconstructed data of Q 3D map descriptors; and search performing the operation of stage r on the reconstructed data of Q 3D map descriptors based on the partial data or all data of the descriptors to obtain N 3D map descriptors. N < Q < P, and the expansion process of the compressed data of P 3D map descriptors includes the expansion of stage m (stage-by-stage expansion) and the expansion of stage k (expansion of the stage), where r and k are positive integers, m < k, and j < r ≤ L. search

[0021] In the present embodiment, search performing the operation of stage i on the binary data of S 3D map descriptors based on the binary data of the descriptors to obtain P 3D map descriptors through screening, performing the expansion of stage m (stage-by-stage expansion) on the compressed data of P 3D map descriptors to obtain the reconstructed data of P 3D map descriptors, search search ​​Based on the partial or full data of the descriptors, the reconstruction data of P 3D map descriptors is search to obtain Q 3D map descriptors through screening; then, perform stage k expansion (stage expansion) on the compressed data of the Q 3D map descriptors to obtain reconstructed data of the Q 3D map descriptors; search Based on all the descriptor data, the reconstruction data of Q 3D map descriptors is search The binary data in the compressed data is processed in at least three stages to obtain N 3D map descriptors. The N 3D map descriptors can be used for positioning. search One stage in search By using search To improve speed, the reconstruction data obtained through different deployments is used for the other two stages. search By using search While ensuring accuracy search Improve speed.

[0022] Specifically, the reconstructed data obtained through the unfolding of stage m and the reconstructed data obtained through the unfolding of stage k have different degrees of unfolding or distortion.

[0023] In one possible design, the compressed data for the P 3D map descriptors includes binary data and quantized data for each of the P 3D map descriptors, and performing stage m of decompression on the compressed data of the P 3D map descriptors to obtain reconstructed data for the P 3D map descriptors includes performing dequantization on the quantized data for each of the P 3D map descriptors to obtain P dequantized data, where the P dequantized data is used as the reconstructed data for the P 3D map descriptors. Performing stage k of decompression on the compressed data of the Q 3D map descriptors to obtain reconstructed data for the Q 3D map descriptors includes performing dequantization on the quantized data for each of the Q 3D map descriptors to obtain Q dequantized data; and obtaining reconstructed data for each of the Q 3D map descriptors based on the Q dequantized data and the binary data for each of the Q 3D map descriptors.

[0024] In this embodiment, dequantization is performed on the quantized data of each of the P 3D map descriptors to obtain dequantized data of each of the P 3D map descriptors, and the dequantized data of each of the P 3D map descriptors (of the current stage) is obtained. search of stage j based on the dequantized data) search to obtain Q 3D map descriptors through screening, perform dequantization on the quantized data of each of the Q 3D map descriptors to obtain Q dequantized data, and obtain reconstructed data for each of the Q 3D map descriptors based on the Q dequantized data and the binary data of the Q 3D map descriptors, and perform stage r based on the reconstructed data for each of the Q 3D map descriptors. search Execute the following to obtain N 3D map descriptors. search The distortion of the reconstructed data for stage r search The distortion of the reconstructed data is larger than that of the coarse search First run search Improve speed, then fine search Run searchImprove accuracy.

[0025] In a possible design, when N=Q, the compressed data of the P 3D map descriptors includes binary data and quantized data of each of the P 3D map descriptors, and performing stage m decompression on the compressed data of the P 3D map descriptors to obtain reconstructed data of the P 3D map descriptors includes: performing dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data; and obtaining the reconstructed data of each of the P 3D map descriptors based on the P dequantized data and the binary data of each of the P 3D map descriptors.

[0026] In this embodiment, the stage j search is a multi-stage search This may be the final stage in search is based on the reconstruction data obtained through full deployment, search Improve accuracy.

[0027] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor; search Based on the binary data of the descriptors, S 3D maps are generated for stage i. search to obtain P 3D map descriptors by: search Based on the binary data of the descriptors, S representative 3D maps are generated for stage i. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0028] The 3D map descriptors in each dataset have a correlation or similarity. A representative 3D map descriptor represents common or similar features of the 3D map descriptors in the corresponding dataset. P can be greater than S, and P <T、S<Tである。

[0029] In this embodiment, one stage of the 3D map descriptor is used for the less representative 3D map descriptor. search Run search Next, we perform one-stage or multi-stage algorithms on datasets corresponding to representative 3D map descriptors. search Run search This can improve the accuracy of the reconstruction data of the 3D map obtained through full unfolding. search When compared with search Improved speed, search Accuracy can be guaranteed.

[0030] In a possible design, Q <P<S<Tである。すなわち、マルチステージの search The number of 3D map descriptors in decreases with each stage.

[0031] In a possible design, the method further comprises: search Based on each component of the descriptor and the corresponding component of the preset threshold vector, search determining a magnitude relationship between each component of the descriptor and a corresponding component of the preset threshold vector; and performing binarization on the magnitude relationship, search The method may comprise obtaining binary data of the descriptor.

[0032] In a possible design, search Based on each component of the descriptor and the corresponding component of the preset threshold vector, search Determining the magnitude relationship between each component of the descriptor and the corresponding component of the preset threshold vector is performed by determining the magnitude relationship between the corresponding component of the preset threshold vector and the corresponding component of the preset threshold vector. searchSubtracting from each component of the descriptor to obtain a difference value for each component, and determining a symbol of each component based on the difference value for each component. Performing binarization with respect to the magnitude relationship, search Obtaining the binary data of the descriptor includes performing binarization on the symbol of each component, search and obtaining the binary data of the descriptor.

[0033] According to a second aspect, an embodiment of the present application provides a method for search for a 3D map. The method includes extracting first binary data of S 3D map descriptors from compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map; search Based on the first binary data of the descriptors, performing search at stage i in the first binary data of the S 3D map descriptors to obtain P 3D map descriptors, where search the descriptors are features extracted from visual information collected by a sensor of an electronic device corresponding to a real environment; extracting second binary data of the P 3D map descriptors from compressed data of the P 3D map descriptors; and search Based on the second binary data of the descriptors, performing search at stage j in the second binary data of the P 3D map descriptors to obtain Q 3D map descriptors, where S, P, Q, i, and j are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents search the total number of stages of the 3D map or search a stage number threshold, and L is a positive integer greater than 1.

[0034] In this embodiment, first binary data of S 3D map descriptors is extracted from compressed data of the S 3D map descriptors, search and based on the first binary data of the descriptors, performing at stage i in the first binary data of the S 3D map descriptors, search It can be seen that by executing search , P 3D map descriptors are obtained through screening. Extract the second binary data of the P 3D map descriptors from the compressed data of the P 3D map descriptors, search Based on the second binary data of the descriptors, for the second binary data of the P 3D map descriptors, at stage j search Execute search to obtain fewer 3D map descriptors through screening. Extract the binary data of the 3D map descriptors from the compressed data for one stage of search Alternately execute with one stage of search Execute at least two stages of search in the 3D map in the compressed state, thereby search Obtain the 3D map descriptors obtained after search The information content retained in the binary data of the 3D map descriptors used by each of any two stages of search is different, or the information integrity of the binary data of the 3D map descriptors used by each of any two stages of search is different. Thus, when comparing with search in the reconstructed data of the 3D map obtained through full expansion, search The speed can be improved and

[0035] The first binary data and the second binary data can be the binary data used in any two stages in the multi-stage search and the binaryization method or length of the binary data and the second binary data can be different.

[0036] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors, N is a positive integer, and 0 < N ≤ Q. The method further includes a step of performing positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0037] In a possible design, both search used in stage i search and in stage j search are based on a first distance search where P 3D map descriptors belong to S 3D map descriptors, and the position of the first binary data of each 3D map descriptor among the P 3D map descriptors in the compressed data of the 3D map descriptors is different from the position of the second binary data of the 3D map descriptors in the compressed data of the 3D map descriptors, and P < S. For example, the position can be a starting position or an ending position or the like.

[0038] search speed and search accuracy are comprehensively improved through the distinction in at least two stages search as can be seen.

[0039] In a possible design, the length of the first binary data of each 3D map descriptor among the P 3D map descriptors is smaller than the length of the second binary data of each 3D map descriptor among the P 3D map descriptors.

[0040] The length of the first binary data of each 3D map descriptor in search stage i is smaller, search improving speed, and the length of the second binary data of each 3D map descriptor in search stage j is larger, search improving accuracy as can be seen. search speed and search accuracy are comprehensively improved through the distinction in at least two stages search as can be seen.

[0041] In a possible design, the method further comprises search a stage of receiving a descriptor, and search a stage of performing binarization on the descriptor to search obtain the first binary data and the second binary data of the descriptor. In the second aspect searchThe executing entity (which may be an electronic device or may be a server (e.g., a cloud server)) configured to execute the method receives the data transmitted by another device. search For example, in the second aspect, search The subject of the method is configured to perform the method. ... search Receive the descriptor. Then, search Perform binarization on the descriptors, search The first binary data and the second binary data of the descriptor are obtained.

[0042] In one possible design, the method further comprises receiving visual information; search extracting descriptors from the visual information; and search Perform binarization on the descriptors, search The method further comprises obtaining first and second binary data of the descriptor. search An executing entity (which may be an electronic device or a server) configured to execute the method receives visual information transmitted by another device, search Descriptors can then be extracted from the visual information. search Perform binarization on the descriptors, search The first binary data and the second binary data of the descriptor are obtained.

[0043] In a possible design, the method further comprises the steps of: triggering the sensor to perform visual information gathering on the real environment in response to a visual information gathering operation input by a user to acquire visual information; search extracting descriptors from the visual information; and search Perform binarization on the descriptors, search The method further comprises obtaining first and second binary data of the descriptor. search An entity (which may be an electronic device or a server) configured to perform the method collects visual information; search Descriptors can then be extracted from the visual information. searchPerform binarization on the descriptors, search The first binary data and the second binary data of the descriptor are obtained.

[0044] In a possible design, search the length of the first binary data of the descriptor is equal to the length of the first binary data of each 3D map descriptor of the S 3D map descriptors; and / or search The length of the second binary data of the descriptor is equal to the length of the second binary data of each 3D map descriptor of the S 3D map descriptors.

[0045] In this embodiment, a first distance between each piece of binary data having the same length is calculated to form a 3D map. search , which results in search Improve speed.

[0046] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor; search In stage i, the first binary data of the S 3D map descriptors is calculated based on the first binary data of the descriptors. search To obtain P 3D map descriptors, search In stage i, the first binary data of the S representative 3D map descriptors is calculated based on the first binary data of the descriptors. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0047] In a possible design, Q <P<S<Tである。すなわち、マルチステージの search The number of 3D map descriptors in decreases with each stage.

[0048] According to a third aspect, an embodiment of the present application provides a method for search for generating a 3D map. The method includes: performing an unfolding at stage m on compressed data of S 3D map descriptors to obtain first reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map; search performing an operation at stage i on the first reconstructed data of the S 3D map descriptors based on partial data or all data of the descriptors to obtain P 3D map descriptors, where search the descriptors are features extracted from visual information collected by sensors of an electronic device corresponding to a real environment; performing an unfolding at stage k on compressed data of the P 3D map descriptors to obtain second reconstructed data of the P 3D map descriptors, where the unfolding process of the compressed data of the S 3D map descriptors includes the unfolding at stage m and the unfolding at stage k, or the unfolding process of the compressed data of the S 3D map descriptors includes the unfolding at stage k, and the unfolding at stage k includes the unfolding at stage m; and search performing an operation at stage j on the second reconstructed data of the P 3D map descriptors based on partial data or all data of the descriptors to obtain Q 3D map descriptors, where S, P, Q, i, j, k, and m are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents search the total number of stages of the 3D map or search a stage number threshold, and L is a positive integer greater than 1. search In this embodiment, an unfolding at stage m is performed on the compressed data of the S 3D map descriptors to obtain the first reconstructed data of the S 3D map descriptors, and search an operation at stage i is performed on the first reconstructed data of the S 3D map descriptors based on partial data or all data of the descriptors.

[0049] search search ​​It can be seen that by executing, P 3D map descriptors are obtained through screening. The expansion of stage k is executed on the compressed data of the P 3D map descriptors to obtain the second reconstructed data of the P 3D map descriptors, search Based on the partial data or all data of the descriptors, in the second reconstructed data of the P 3D map descriptors at stage j search is executed to obtain fewer 3D map descriptors through screening. After expansion, the reconstructed data of the 3D map descriptors for one stage of search is obtained, and this is executed alternately with one stage of search For at least two stages of search in one stage of search is a staged expansion, search which improves the speed. For at least two stages of search the differentiated expansion settings for search can guarantee the accuracy. The reconstructed data of the 3D map descriptors used by any two stages of search has different degrees of expansion or distortion. Thus, when compared with search in the reconstructed data of the 3D map obtained through full expansion, search the speed is improved, search and the accuracy can be guaranteed.

[0050] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors, N is a positive integer, and 0 < N ≤ Q. The method further includes the step of performing positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0051] In this embodiment, search the 3D map points corresponding to the N 3D map descriptors obtained through searchWhen the map points corresponding to the descriptors match, positioning can be performed based on the 3D map points corresponding to the N 3D map descriptors, which helps improve positioning performance.

[0052] In a possible design, stage i search and stage j search Both used in search The method is based on the second distance search The method includes: P 3D map descriptors belonging to S 3D map descriptors; a distortion of a first reconstructed data of each of the P 3D map descriptors is different from a distortion of a second reconstructed data of each of the P 3D map descriptors; the distortion of the first reconstructed data of each of the P 3D map descriptors indicates a degree of difference between the first reconstructed data of each of the P 3D map descriptors and the corresponding original 3D map descriptor; and the distortion of the second reconstructed data of each of the P 3D map descriptors indicates a degree of difference between the second reconstructed data of each of the P 3D map descriptors and the corresponding original 3D map descriptor, where P <Sである。

[0053] search Speed and search Accuracy is at least two stages search It can be seen that the distinction between the two improves the overall performance.

[0054] In a possible design, the distortion of the first reconstructed data for each 3D map descriptor of the P 3D map descriptors is greater than the distortion of the second reconstructed data for each 3D map descriptor of the P 3D map descriptors.

[0055] Stage I search It can be seen that the distortion of the first reconstructed data of each 3D map descriptor in P can be greater than the distortion of the second reconstructed data of each 3D map descriptor in P 3D map descriptors. search In the previous stage search The distortion of the next stage search distortion is greater than that of the previous stage. search High speed searchImplementing search Improve speed and next stage search In detail search Implementing search Improve accuracy.

[0056] In a possible design, the method further comprises: search receiving a descriptor; and search The method further comprises a step of acquiring partial or all data of the descriptor. search The executing entity (which may be an electronic device or may be a server (e.g., a cloud server)) configured to execute the method receives the data transmitted by another device. search For example, in the third aspect, search The subject of the method is configured to perform the method. ... search Receive the descriptor. Then, search Partial or all data of the descriptor search Obtained as per requirements.

[0057] In one possible design, the method further comprises receiving visual information; search extracting descriptors from the visual information; and search The method further comprises a step of acquiring partial or all data of the descriptor. search An executing entity (which may be an electronic device or a server) configured to execute the method receives visual information transmitted by another device, search Descriptors can then be extracted from the visual information. search Partial or all data of the descriptor search Obtained as per requirements.

[0058] In a possible design, the method further comprises the steps of: triggering the sensor to perform visual information gathering on the real environment in response to a visual information gathering operation input by a user to acquire visual information; search extracting descriptors from the visual information; and searchThe method further comprises a step of acquiring partial or all data of the descriptor. search An entity (which may be an electronic device or a server) configured to perform the method collects visual information; search Descriptors can then be extracted from the visual information. search Partial or all data of the descriptor search Obtained as per requirements.

[0059] In this embodiment of the present application, the 3D map search for According to the method for search in different application scenarios search It can be performed on 3D maps based on descriptors, which is widely applicable.

[0060] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor; search In the first reconstruction data of S 3D map descriptors based on partial data or all data of the descriptors, search To obtain P 3D map descriptors, search Based on the partial or full data of the descriptors, the first reconstruction data of S representative 3D map descriptors is performed in stage i. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0061] The 3D map descriptors in each dataset have a correlation or similarity. A representative 3D map descriptor represents common or similar features of the 3D map descriptors in the corresponding dataset. P can be greater than S, and P <T、S<Tである。

[0062] In this embodiment, in a smaller number of representative 3D map descriptors, perform one stage of search to search improve the speed. Next, perform one stage or multiple stages of search on a dataset corresponding to the representative 3D map descriptors, search which may improve the accuracy. Thereby, when compared with search in the reconstructed data of the 3D map obtained through full expansion, search the speed can be improved and search the accuracy can be guaranteed.

[0063] In a possible design, Q < P < S < T. That is, the number of 3D map descriptors in the multi-stage search decreases for each stage.

[0064] According to the fourth aspect, an embodiment of the present application provides a method for search for a 3D map. The method includes: performing an expansion of stage m on the compressed data of S 3D map descriptors to obtain reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map; search performing search of stage i on the reconstructed data of the S 3D map descriptors based on partial data or all data of the descriptors to obtain P 3D map descriptors, where search the descriptors are features extracted from visual information collected by a sensor of an electronic device corresponding to the real environment; extracting binary data of the P 3D map descriptors from the compressed data of the P 3D map descriptors; and search performing search of stage j on the binary data of the P 3D map descriptors based on the binary data of the descriptors to obtain Q 3D map descriptors, where S, P, Q, i, m, and j are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L is the searchThe total number of stages or search a stage number threshold, where L is a positive integer greater than 1.

[0065] In this embodiment, at least two stages of search in one stage of search is expanded to search improve the accuracy, and at least two stages of search in one stage of search is extracted from the compressed data to search improve the speed. It can be seen that compared with the search in the reconstructed data of the 3D map obtained through full expansion, the 3D map provided in the present embodiment of the present application search for The method for search can improve the speed and search can guarantee the accuracy.

[0066] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors, N is a positive integer, and 0 < N ≤ Q. The method further includes a step of performing positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0067] In a possible design, the search used for search the method in stage i is a search method based on the second distance, and the search used for search the method in stage j is a search method based on the first distance.

[0068] In this embodiment, the search method based on the second distance search can improve the accuracy, and the search method based on the first distance search can improve the speed, thereby search comprehensively improving the accuracy and search the speed.

[0069] In a possible design, the method further comprises: search receiving a descriptor; and search Perform binarization on the descriptor search The method further comprises the step of obtaining binary data of the descriptor. search The executing entity (which may be an electronic device or may be a server (e.g., a cloud server)) configured to execute the method receives the data transmitted by another device. search For example, in the first aspect, search The subject of the method is configured to perform the method. ... search Receive the descriptor. Then, search Perform binarization on the descriptors, search Gets the binary data of the descriptor.

[0070] In one possible design, the method further comprises receiving visual information; search extracting descriptors from the visual information; and search Perform binarization on the descriptor search The method further comprises the step of obtaining binary data of the descriptor. search An executing entity (which may be an electronic device or a server) configured to execute the method receives visual information transmitted by another device, search Descriptors can then be extracted from the visual information. search Perform binarization on the descriptors, search Gets the binary data of the descriptor.

[0071] In a possible design, the method further comprises the steps of: triggering the sensor to perform visual information gathering on the real environment in response to a visual information gathering operation input by a user to acquire visual information; search extracting descriptors from the visual information; and search Perform binarization on the descriptor search The method further comprises the step of obtaining binary data of the descriptor. search An executing entity (which may be an electronic device or a server) configured to execute the method receives visual information transmitted by another device, search Descriptors can then be extracted from the visual information. search Perform binarization on the descriptors, search Gets the binary data of the descriptor.

[0072] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor; search In the reconstruction data of S 3D map descriptors based on partial or full data of the descriptors, search To obtain P 3D map descriptors, search Based on the partial or full data of the descriptors, the reconstruction data of S representative 3D map descriptors is performed in stage i. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0073] The 3D map descriptors in each dataset have a correlation or similarity. A representative 3D map descriptor represents common or similar features of the 3D map descriptors in the corresponding dataset. P can be greater than S, and P <T、S<Tである。

[0074] In this embodiment, one stage of the 3D map descriptor is used for the less representative 3D map descriptor. search Run search Next, we perform one-stage or multi-stage algorithms on datasets corresponding to representative 3D map descriptors. search Run searchThis can improve the accuracy of the reconstruction data of the 3D map obtained through full unfolding. search When compared with search Improved speed, search Accuracy can be guaranteed.

[0075] In a possible design, Q <P<S<Tである。すなわち、マルチステージの search The number of 3D map descriptors in decreases with each stage.

[0076] With reference to the first aspect or any possible design of the first aspect, or with reference to the second aspect or any possible design of the second aspect, or with reference to the third aspect or any possible design of the third aspect, or with reference to the fourth aspect or any possible design of the fourth aspect, there are further the following possible designs:

[0077] In a possible design, search The descriptors include area descriptors or 3D map point descriptors. Accordingly, the S 3D map descriptors may include S area descriptors or S 3D map point descriptors.

[0078] According to a fifth aspect, an embodiment of the present application provides a 3D map. search for The apparatus may be a chip or system-on-chip in an electronic device or server, or may be a functional module in an electronic device or server configured to implement the method according to the first aspect or any possible implementation of the first aspect. search for The device for search The module includes an extraction module and an extraction module.

[0079] In a possible design, the extraction module is configured to extract binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map. search The module is searchBased on the binary data of the descriptors, it is configured to execute search at stage i in the binary data of S 3D map descriptors to obtain P 3D map descriptors. Here, search the descriptor is a feature extracted from visual information collected by a sensor of an electronic device corresponding to the real environment. The expansion module is configured to execute the expansion at stage m on the compressed data of the P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors. Here, the expansion process of the compressed data of the P 3D map descriptors includes at least the expansion at stage m. search Based on partial or all of the data of the descriptors, it is configured to execute search at stage j in the reconstructed data of the P 3D map descriptors to obtain Q 3D map descriptors. Here, S, P, Q, i, j, and m are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, where T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the search total number of stages of the 3D map or search the stage number threshold of the 3D map, and L is a positive integer greater than 1. search In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors. N is a positive integer, 0 < N ≤ Q, and the device further includes a pose determination module. The pose determination module executes positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device. search In a possible design, the method used for search at stage i is a

[0080] method based on the first distance, and the method used for search at stage j is a search method based on the second distance.

[0081] search used for search search search search used for search

[0082] search search

[0082] In a possible design, the apparatus further: search receives a descriptor, search performs binarization on the descriptor to search obtain the binary data of the descriptor; or receives visual information and extracts search a descriptor from the visual information, search performs binarization on the descriptor to search obtain the binary data of the descriptor; or in response to a visual information collection operation input by a user, triggers a sensor to perform visual information collection on the real environment to obtain visual information, search extracts a descriptor from the visual information, search performs binarization on the descriptor to search obtain the binary data of the descriptor, and is configured with an acquisition module.

[0083] In a possible design, when N < Q, the expansion module further performs the expansion at stage k on the compressed data of Q 3D map descriptors to obtain the reconstructed data of Q 3D map descriptors; and search performs at stage r in the reconstructed data of Q 3D map descriptors based on the partial data or all data of the descriptor to search obtain N 3D map descriptors, where N < Q < P, and the expansion process of the compressed data of P 3D map descriptors includes the expansion at stage m and the expansion at stage k, where r and k are positive integers, m < k, j < r ≦ L, and is configured to perform the above.

[0084] In a possible design, the compressed data of the P 3D map descriptors includes binary data and quantized data of each of the P 3D map descriptors, and the decompression module is specifically configured to: perform dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data, where the P dequantized data are used as reconstruction data for the P 3D map descriptors; perform dequantization on the quantized data of each of the Q 3D map descriptors to obtain Q dequantized data; and obtain reconstruction data for each of the Q 3D map descriptors based on the Q dequantized data and the binary data of each of the Q 3D map descriptors.

[0085] In a possible design, when N=Q, the compressed data of the P 3D map descriptors includes binary data and quantized data of each of the P 3D map descriptors; the decompression module is specifically configured to: perform dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data; and obtain reconstructed data of each of the P 3D map descriptors based on the P dequantized data and the binary data of each of the P 3D map descriptors.

[0086] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor; search The modules are: search Based on the binary data of the descriptor, S representative 3D maps are generated for stage i in the binary data of the descriptor. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0087] According to a sixth aspect, an embodiment of the present application provides a 3D map. search for To provide an apparatus. The apparatus may be a chip or a system-on-chip in an electronic device or a server, or a functional module in an electronic device or a server configured to implement a method according to the first aspect or any possible implementation of the first aspect. For example, a 3D map search for The apparatus for search Includes a module, an extraction module, and a deployment module.

[0088] In a possible design, the extraction module is configured to extract the first binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map. search The module search Based on the first binary data of the descriptors, perform search At stage i in the first binary data of the S 3D map descriptors to obtain P 3D map descriptors, where search The descriptors are features extracted from visual information collected by sensors of an electronic device corresponding to the real environment. The extraction module is further configured to extract the second binary data of the P 3D map descriptors from the compressed data of the P 3D map descriptors. search The module search Based on the second binary data of the descriptors, perform search At stage j in the second binary data of the P 3D map descriptors to obtain Q 3D map descriptors, where S, P, Q, i, and j are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents search The total number of stages or search The stage number threshold of the 3D map, and L is a positive integer greater than 1.

[0089] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors. N is a positive integer, 0 < N ≤ Q, and the apparatus further includes a pose determination module. The pose determination module is configured to perform positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain pose information of the electronic device.

[0090] In a possible design, both search used for search stage i and search stage j search are based on the first distance. P of the 3D map descriptors belong to S of the 3D map descriptors. The position of the first binary data of each of the P 3D map descriptors in the compressed data of the 3D map descriptors is different from the position of the second binary data of the 3D map descriptors in the compressed data of the 3D map descriptors, and P < S.

[0091] In a possible design, the length of the first binary data of each of the P 3D map descriptors is smaller than the length of the second binary data of each of the P 3D map descriptors.

[0092] In a possible design, the apparatus further: search receives descriptors, search performs binarization on the descriptors to search obtain the first binary data and the second binary data of the descriptors; or receives visual information, extracts search descriptors from the visual information, search performs binarization on the descriptors to search obtain the first binary data and the second binary data of the descriptors; or triggers a sensor to collect visual information for the real environment in response to a visual information collection operation input by the user, search extracts descriptors from the visual information, search performs binarization on the descriptors to search An acquisition module configured to acquire the first binary data and the second binary data of the descriptor.

[0093] In a possible design, search the length of the first binary data of the descriptor is equal to the length of the first binary data of each 3D map descriptor of the S 3D map descriptors; and / or search The length of the second binary data of the descriptor is equal to the length of the second binary data of each 3D map descriptor of the S 3D map descriptors.

[0094] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor; search Specifically, the module is search In stage i, the first binary data of the S representative 3D map descriptors is calculated based on the first binary data of the descriptors. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0095] According to a seventh aspect, an embodiment of the present application provides a 3D map. search for The apparatus may be a chip or system-on-chip in an electronic device or server, or may be a functional module in an electronic device or server configured to implement the method according to the first aspect or any possible implementation of the first aspect. search for The device for search The module includes an extraction module and an extraction module.

[0096] In a possible design, the expansion module is configured to perform the expansion of stage m on the compressed data of S 3D map descriptors to obtain the first reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map. search The module is search Based on the partial data or all data of the descriptors, perform the search of stage i in the first reconstructed data of the S 3D map descriptors to obtain P 3D map descriptors, where search The descriptors are features corresponding to the real environment, extracted from the visual information collected by the sensors of the electronic device. The expansion module is further configured to perform the expansion of stage k on the compressed data of the P 3D map descriptors to obtain the second reconstructed data of the P 3D map descriptors, where the expansion process of the compressed data of the S 3D map descriptors includes the expansion of stage m and the expansion of stage k, or the expansion process of the compressed data of the S 3D map descriptors includes the expansion of stage k, and the expansion of stage k includes the expansion of stage m. search The module is search Based on some or all of the descriptors, perform the search of stage j in the second reconstructed data of the P 3D map descriptors to obtain Q 3D map descriptors, where S, P, Q, i, j, k, and m are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L is the search total number of stages of the 3D map or search the stage number threshold, and L is a positive integer greater than 1.

[0097] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors are searchIt coincides with the 3D map points corresponding to the descriptors, N is a positive integer, 0 < N ≤ Q, and the device further includes a pose determination module. The pose determination module is configured to perform positioning based on the 3D map points corresponding to N 3D map descriptors to obtain the pose information of the electronic device.

[0098] In a possible design, for stage i search and stage j search both used in search The method is a method based on the second distance. P 3D map descriptors belong to S 3D map descriptors, and the distortion of the first reconstruction data of each 3D map descriptor among the P 3D map descriptors is different from the distortion of the second reconstruction data of each 3D map descriptor among the P 3D map descriptors. The distortion of the first reconstruction data of each 3D map descriptor among the P 3D map descriptors indicates the degree of difference between the first reconstruction data of each 3D map descriptor and the corresponding original 3D map descriptor, and the distortion of the second reconstruction data of each 3D map descriptor among the P 3D map descriptors indicates the degree of difference between the second reconstruction data of each 3D map descriptor and the corresponding original 3D map descriptor, where P < S.

[0099] In a possible design, the distortion of the first reconstruction data of each 3D map descriptor among the P 3D map descriptors is greater than the distortion of the second reconstruction data of each 3D map descriptor among the P 3D map descriptors.

[0100] In a possible design, the device further search receives the descriptors, search obtains partial data or all data of the descriptors; or receives visual information, search extracts the descriptors from the visual information, search obtains partial data or all data of the descriptors; or, in response to a visual information collection operation input by the user, triggers a sensor to perform visual information collection on the real environment to obtain visual information, search extracts the descriptors from the visual information and search ​​It includes an acquisition module configured to acquire partial or full data of the descriptor.

[0101] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor; search Specifically, the module is search Based on the partial or full data of the descriptors, the m-th reconstruction data of the S representative 3D map descriptors is obtained in the stage i. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0102] According to an eighth aspect, an embodiment of the present application provides a 3D map. search for The apparatus may be a chip or system-on-chip in an electronic device or server, or may be a functional module in an electronic device or server configured to implement the method according to the first aspect or any possible implementation of the first aspect. search for The device for search The module includes an extraction module and an extraction module.

[0103] In a possible design, the decompression module is configured to perform stage m decompression on the compressed data of the S 3D map descriptors to obtain reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map. search The module is search Based on the partial or full data of the descriptors, the reconstruction data of S 3D map descriptors is search to obtain P 3D map descriptors, where: searchA descriptor is a feature corresponding to the real environment, extracted from visual information collected by a sensor of an electronic device. The extraction module is configured to extract binary data of P descriptors from the compressed data of P 3D map descriptors. search The module is further configured to search perform, based on the binary data of the descriptors, search at stage j in the binary data of P 3D map descriptors to obtain Q 3D map descriptors, where S, P, Q, i, m, and j are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, and 1 < j ≤ L, and L represents the search total number of stages or search stage number threshold of the 3D map, and L is a positive integer greater than 1.

[0104] In a possible design, N of the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors, N is a positive integer, 0 < N ≤ Q, and the device further includes a pose determination module. The pose determination module is configured to perform positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0105] In a possible design, the search method used for search stage i is a search method based on the second distance, and the search method used for search stage j is a search method based on the first distance.

[0106] In a possible design, the device further: search receives a descriptor, search performs binarization on the descriptor to search obtain the binary data of the descriptor; or receives visual information and extracts from the visual information search Extract the descriptors, search Perform binarization on the descriptors, search Acquire binary data of the descriptor; or trigger a sensor to perform visual information acquisition on the real environment in response to a visual information acquisition operation input by a user to acquire visual information; search Descriptors are extracted from visual information, search Perform binarization on the descriptor search An acquisition module configured to acquire binary data of the descriptor is provided.

[0107] In a possible design, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor; search Specifically, the module is search Based on the partial or full data of the descriptors, the m-th reconstruction data of the S representative 3D map descriptors is obtained in the stage i. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0108] According to a ninth aspect, an embodiment of the present application provides a method for generating a 3D map, the method comprising one or more processors and a memory configured to store one or more programs. search for When the one or more programs are executed by the one or more processors, the one or more processors are capable of implementing a method according to any one of the first to fourth aspects or any possible design of the first to fourth aspects.

[0109] According to a tenth aspect, an embodiment of the present application provides a computer-readable storage medium comprising a computer program, which, when run on a computer, enables the computer to perform a method according to any one of the first to fourth aspects or any possible design thereof.

[0110] According to an eleventh aspect, embodiments of the present application provide a computer program or a computer program product, which, when run on a computer, enables the computer to implement any one of the methods described in the first to fourth aspects or any possible implementations of the first to fourth aspects.

[0111] It should be understood that the technical solutions in the fifth to eleventh aspects of the present application are consistent with the technical solutions in the first to fourth aspects of the present application, and the beneficial effects achieved in the aspects and corresponding feasible implementations are similar thereto, and the details will not be described again. [Brief explanation of the drawings]

[0112] [Figure 1] FIG. 1 is a schematic diagram of an application architecture according to an embodiment of the present application.

[0113] [Figure 2] 1 is a schematic diagram of the structure of an electronic device 20 according to an embodiment of the present application.

[0114] [Figure 3] 2 is a schematic diagram of the structure of a server 30 according to an embodiment of the present application.

[0115] [Figure 4a] 1A-1D are schematic diagrams of various application scenarios according to embodiments of the present application; [Figure 4b] 1A-1D are schematic diagrams of various application scenarios according to embodiments of the present application; [Figure 4c]1A-1D are schematic diagrams of various application scenarios according to embodiments of the present application; [Figure 4d] 1A-1D are schematic diagrams of various application scenarios according to embodiments of the present application; [Figure 4e] 1A-1D are schematic diagrams of various application scenarios according to embodiments of the present application; [Figure 4f] 1A-1D are schematic diagrams of various application scenarios according to embodiments of the present application;

[0116] [Figure 4g] 1 is a schematic diagram of a user interface (e.g., a user interface of a 3D map application) displayed by an electronic device according to an embodiment of the present application.

[0117] [Figure 5A] 2 is a schematic flowchart of a segment of a method for searching a 3D map according to an embodiment of the present application; [Figure 5B] 2 is a schematic flowchart of a segment of a method for searching a 3D map according to an embodiment of the present application; [Figure 5C] 2 is a schematic flowchart of a segment of a method for searching a 3D map according to an embodiment of the present application; [Figure 5D] 2 is a schematic flowchart of a segment of a method for searching a 3D map according to an embodiment of the present application;

[0118] [Figure 6] 1 is a schematic flowchart of a method for searching a 3D map according to an embodiment of the present application;

[0119] [Figure 7] 1 is a schematic flowchart of a method for searching a 3D map according to an embodiment of the present application;

[0120] [Figure 8] 1 is a schematic flowchart of a method for searching a 3D map according to an embodiment of the present application;

[0121] [Figure 9] 2 is a schematic diagram of a processing process of a method for searching a 3D map according to an embodiment of the present application;

[0122] [Figure 10] 1 is a schematic diagram of the structure of an apparatus for searching a 3D map according to an embodiment of the present application;

[0123] [Figure 11] 11 is a schematic block diagram of a decoding device 1100 according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0124] Hereinafter, the embodiments of the present application will be described with reference to the accompanying drawings of the embodiments of the present application. The terms used in the embodiments of the present application are only used to describe specific embodiments of the present application and are not intended to limit the present application.

[0125] In the embodiments, claims, and accompanying drawings of this specification, terms such as "first," "second," etc. are intended to distinguish and describe only and are not to be understood as indicating or suggesting relative importance or sequence. Furthermore, the terms "comprise," "have," and any variations thereof are intended to cover non-exclusive inclusions, e.g., the inclusion of a series of steps or units. A method, system, product, or device is not necessarily limited to explicitly recited steps or units, but may include other steps or units that are not explicitly recited and are inherent to the process, method, product, or device.

[0126] In this application, "at least one (item)" should be understood to mean one or more, and "multiple" should be understood to mean two or more. The term "and / or" describes an association relationship between associated objects and indicates that three relationships may exist. For example, "A and / or B" may indicate three cases: only A is present, only B is present, and both A and B are present. A and B may be singular or plural. The symbol " / " typically indicates an "or" relationship between associated objects. "At least one of the following items" or similar expressions indicates any combination of the items, including any combination of one of the items or any combination of more than one of the items. For example, "at least one of a, b, or c" may refer to a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c may be singular or plural.

[0127] In the embodiments of the present application, terms such as "example" or "for example" are used to mean serving as an example, illustration, or explanation. Any embodiment or design described in the embodiments of the present application by an "example" or "for example" is not to be construed as preferred or advantageous over other embodiments or designs. Rather, terms such as "example" or "for example" are intended to present a relative perspective in a particular manner.

[0128] As used in the embodiments of this application and the appended claims, the singular forms "a," "said," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should be further understood that the term "and / or," as used herein, denotes and includes any or all possible combinations of one or more of the associated listed items. As used herein, the meaning of the singular forms "a," "some," and "the" also includes the plural forms, unless the context clearly dictates otherwise. It should be further understood that the terms "comprise," "have," "comprise," and / or "include" are used herein to specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0129] It should be noted that the terminology used in the embodiments of the present application is for the purpose of illustrating particular embodiments only and is not intended to be limiting of the present application.

[0130] FIG. 1 is a schematic diagram of an application architecture according to an embodiment of the present application. As shown in FIG. 1, the application architecture includes a plurality of electronic devices and a server. The plurality of electronic devices may include a first electronic device and one or more second electronic devices (two second electronic devices are used as an example in FIG. 1). The one or more second electronic devices are a plurality of electronic devices other than the first electronic device. Communication may be performed between the plurality of electronic devices and the server, and among the plurality of electronic devices. For example, any device in the application architecture may communicate with another device via wireless fidelity (Wi-Fi®) communication, Bluetooth® communication, or cellular second / third / fourth / fifth generation (2G / 3G / 4G / 5G) communication, etc. It should be understood that other communication methods, including future communication methods, may also be used between the server and the electronic devices. This specification does not particularly limit this. It should be noted that the term "one or more second electronic devices" in this embodiment of the present application is merely used to refer to electronic devices other than the first electronic device, and does not limit whether the multiple electronic devices are of the same type.

[0131] The electronic device may be various types of devices equipped with a camera and a display component. For example, the electronic device may be a terminal device such as a mobile phone, a tablet computer, a notebook computer, or a video recorder (in FIG. 1 , a mobile phone is used as an example of an electronic device). Alternatively, the electronic device may be a device used for interaction in a virtual scenario, including VR glasses, an AR device, an MR interaction device, etc. Alternatively, the electronic device may be a wearable electronic device such as a smart watch or a smart band. Alternatively, the electronic device may be a device carried in a transport device such as a vehicle, an unmanned vehicle, an unmanned aerial vehicle, or an industrial robot. The specific form of the electronic device is not particularly limited in the embodiments of the present application.

[0132] Furthermore, an electronic device may also be referred to as user equipment (UE), a subscriber station, mobile unit, subscriber unit, radio unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, terminal device, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handheld terminal, user agent, mobile client, client, or another suitable terminology.

[0133] The server may be one or more physical servers (in FIG. 1, one physical server is used as an example), or may be a computer cluster, or may be a virtual machine or cloud server in a cloud computing scenario, etc.

[0134] In this embodiment of the present application, a virtual scenario application (APP), such as a VR application, an AR application, or an MR application, may be installed on an electronic device, and the VR application, the AR application, or the MR application may be activated based on a user operation (e.g., tap, touch, slide, shake, or voice control). The electronic device may collect visual information of any object in the environment by using a sensor, and then display a virtual object on a display component based on the collected visual information. The virtual object may be a virtual object (i.e., an object in a virtual environment) in a VR scenario, an AR scenario, or an MR scenario.

[0135] In this embodiment of the present application, a navigation, detection, or control application may be installed on the electronic device, and the corresponding application is activated based on the operation and control of the user or a pre-set program. The electronic device may activate applications such as route planning, object detection, and transport device operation and control based on the pose and other status information of the electronic device in the current environment.

[0136] Pose is the position and orientation information of an electronic device, and may be an absolute pose in the world coordinate system or a relative pose to a point in the environment.

[0137] Visual information in embodiments of the present application includes, but is not limited to, image video collected by a camera (without depth information), image video collected by a depth sensor (with depth information), data collected by a LiDAR, and data collected by a millimeter wave radar (RaDAR).

[0138] It should be noted that in this embodiment of the present application, the virtual scenario application in the electronic device may be an application built into the electronic device, or may be an application provided by a third-party service provider and installed by the user, although this specification does not particularly limit this.

[0139] In this embodiment of the present application, a simultaneous localization and mapping (SLAM) system may be further configured for the electronic device. The SLAM system generates a map of a completely unknown environment and can use this map to perform positioning, pose (position and posture) determination, navigation, etc. In this embodiment of the present application, the map generated by the SLAM system is referred to as a SLAM map. The SLAM map may be understood as a map drawn by the SLAM system based on environmental information collected by a collection device. The collection device may include a visual information collection device and an inertial measurement unit (IMU) in the electronic device. The visual information collection device may include, for example, a camera, a depth camera, a lidar, and a millimeter-wave radar. The IMU may include, for example, sensors such as a gyroscope and an accelerometer.

[0140] In the present embodiment, the SLAM map is also referred to as a 3D map. Note that the 3D map includes, but is not limited to, the SLAM map, and may further include a three-dimensional map generated by using another technique, which is not specifically limited in the present embodiment.

[0141] In a possible implementation, the 3D map may include a plurality of 3D map points, and accordingly, the data of the 3D map may include data of a plurality of 3D map points, which may be points of interest or points having significant features in the environment.

[0142] Possible methods for acquiring 3D map points include using multiple devices, such as lidar, aerial photography (tilt photography) from the field of view of an unmanned aerial vehicle, high-definition panoramic cameras, and high-definition industrial cameras, to perform photography. From the data acquired through photography by the aforementioned devices, 3D map points are extracted using methods such as ORB, scale-invariant feature transform (SIFT), speeded up robust features (SURF), binary robust independent elementary features (BRIEF), binary robust invariant scalable keypoints (BRISK), fast retina keypoint (FREAK), D2Net, or self-supervised training feature point detection and descriptor extraction method (SuperPoint).

[0143] The data for a 3D map point may include:

[0144] [(1) 3D map point descriptor]

[0145] A 3D map point descriptor is a vector used to represent the local features of corresponding 3D map points. In visual positioning algorithms, 3D map point descriptors are used for matching between 3D map points. One possible method is to calculate the distance (which may be Euclidean distance, dot product distance, Hamming distance, etc.) between two 3D map point descriptors; and if the distance is smaller than a threshold, the two 3D map points are considered to match.

[0146] [(2) 3D map point space position]

[0147] The 3D map point spatial position may be represented by using X, Y, and Z on three-dimensional spatial axes, or by using longitude, latitude, and altitude, or by using polar coordinates, etc. The method for representing the 3D map point spatial position is not particularly limited in the embodiments of the present application. The 3D map point spatial position may be the absolute position of the 3D map point, or the relative position of the 3D map point. For example, the center position of the entire area is used as the origin, and all 3D map point spatial positions are positions shifted with respect to the spatial position of the origin.

[0148] In an embodiment of the present application, a number may be assigned to each 3D map point and written into the 3D map data, or the storage order of multiple 3D map points in memory may be used to implicitly indicate the number of the 3D map point. Note that the order of multiple 3D map points included in a 3D map is meaningless. Therefore, the numbers may be considered as identifiers used to identify the 3D map points in order to distinguish them. However, the numbers are not intended to limit the order of the multiple 3D map points. For example, if a 3D map includes three 3D map points numbered 1, 2, and 3, respectively, the three 3D map points may be processed in the order 1, 2, and 3, or 3, 2, and 1, or 2, 1, and 3, etc.

[0149] In a possible implementation, the 3D map data further includes a plurality of area descriptors, any one of which describes characteristics of some or all of the plurality of 3D map points. Specifically, for any one of the plurality of area descriptors, the area descriptor may describe characteristics of some or all of the plurality of 3D map points. In this case, the area descriptor and the 3D map point have a one-to-many relationship. Characteristics of each 3D map point of the plurality of 3D map points may be described by some or all of the area descriptors of the plurality of area descriptors. In this case, the 3D map point and the area descriptor have a one-to-many relationship. It can be seen that the plurality of area descriptors and the plurality of 3D map points have a many-to-many relationship. Methods for generating area descriptors include, but are not limited to, traditional methods such as bag of words (BOW) and vector of locally aggregated descriptors (VLAD), and novel methods based on NetVLAD or artificial intelligence (AI). Similarly, multiple area descriptors may be identified by numbers to distinguish between the multiple area descriptors, but similarly, the numbers are not intended to limit the order of the multiple area descriptors.

[0150] In a possible implementation, the 3D map data further comprises correspondences between 3D map points and descriptors, which clearly describe which 3D map point any descriptor corresponds to, and which 3D map point any descriptor corresponds to.

[0151] Optionally, the aforementioned correspondence may be explicitly described by using a correspondence table between area descriptor numbers and 3D map point numbers. For example, a 3D map includes three area descriptors whose numbers are T1 to T3 and six 3D map points. The numbers of the six 3D map point spatial locations are P1 to P6, and the numbers of the six 3D map point descriptors are F1 to F6. The correspondence table is shown in Table 1. Table 1 [Table 1]

[0152] It should be noted that Table 1 is an example of a correspondence table between area descriptor numbers and 3D map point numbers. The correspondence table may alternatively be presented in another format or manner, which is not specifically limited in this application.

[0153] Optionally, the aforementioned correspondence may alternatively be implicitly described by using the storage locations of the area descriptors and 3D map points. For example, T1 is first stored in memory, then the data of P1, P2, and P3 are stored; then T2 is stored, then the data of P2 and P3 are stored; finally, T3 is stored, then the data of P3, P4, P5, and P6 are stored.

[0154] FIG. 2 is a schematic diagram of the structure of an electronic device 20 according to an embodiment of the present application. As shown in FIG. 2, the electronic device 20 may be at least one of the first electronic device and one or more second electronic devices in the embodiment shown in FIG. 1. It should be understood that the structure shown in FIG. 2 does not constitute a specific limitation on the electronic device 20. In some other embodiments of the present application, the electronic device 20 may include more or fewer components than those shown in FIG. 2, combine some components, separate some components, or have a different component configuration. The components shown in FIG. 2 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0155] The electronic device 20 may include a chip 21, a memory 22 (one or more computer-readable storage media), a user interface 23, a display component 24, a camera 25, sensors 26, a positioning module 27 configured to perform device positioning, and a transceiver 28 configured to perform communications. These components may communicate with each other by using one or more buses 29.

[0156] One or more processors 211, a clock module 212, and a power management module 213 may be integrated into the chip 21. The clock module 212 integrated into the chip 21 is mainly configured to provide timers required for data transmission and timing control of the processor 211. The timers may implement clock functions for data transmission and timing control. The processor 211 may perform operations based on instruction operation codes and timing signals and generate operation control signals to complete control of instruction fetching and instruction execution. The power management module 213 integrated into the chip 21 is mainly configured to provide stable and highly accurate voltages for the chip 21 and other components of the electronic device 20.

[0157] The processor 211 may also be referred to as a central processing unit (CPU). The processor 211 may specifically include one or more processing units. For example, the processor 211 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), etc. The different processing units may be separate components or may be integrated into one or more processors.

[0158] In possible implementations, the processor 211 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, a universal serial bus (USB) port, etc.

[0159] The memory 22 may be connected to the processor 211 via a bus 29 or may be coupled to the processor 311 and configured to store various software programs and / or groups of instructions. The memory 22 may include high-speed random-access memory (e.g., cache) or non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 22 may store an operating system, such as an embedded operating system, such as Android®, Apple Mobile Platform (iOS®), Microsoft Windows® Operating System (Windows®), or Linux®. The memory 22 may further store data, such as image data, point cloud data, 3D map data, pose data, coordinate system transformation information, and map update information. The memory 22 may also store computer-executable program code, including instructions, such as SLAM system communication program instructions and related program instructions. The memory 22 may further store one or more applications, such as a virtual scenario application (e.g., AR / VR / MR), a map application, an image management application, and a navigation and control application. The memory 22 may further store a user interface program. The user interface program may vividly display the content of the application (e.g., virtual objects in a virtual scenario (e.g., AR / VR / MR)) using a graphical operation interface, present the content using the display component 24, and receive control operations performed by a user on the application using input controls such as menus, dialog boxes, or buttons.

[0160] The user interface 23 may be, for example, a touch panel, which may detect an operation command performed by a user on the touch panel, or may be, for example, a keypad, physical buttons, or a mouse.

[0161] The electronic device 20 may include one or more display components 24. The electronic device 20 may implement display functions by using the display component 24, a graphics processing unit (GPU), an application processor (AP), and the like within the chip 21 in cooperation with each other. The GPU is a microprocessor for implementing image processing and is connected to the display component 24 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The display component 24 may display interface content output by the electronic device 20, such as images, videos, and the like in virtual scenarios such as AR / VR / MR. The interface content may include interfaces such as running applications and system-level menus, and specifically may include the following interface elements: input interface elements such as buttons, text entry boxes, scrollbars, and menus; and output interface elements such as windows, labels, images, videos, and animations.

[0162] The display component 24 may be a display panel, a lens (e.g., VR glasses), a projection screen, etc. The display panel may also be referred to as a display screen, and may be, for example, a touch screen, a flexible screen, a curved screen, etc., or may be another optical component. It should be understood that the display screen of the electronic device in the embodiments of the present application may be a touch screen, a flexible screen, a curved screen, or a screen of another form. In other words, the display screen of the electronic device has the function of displaying images, and the specific material and shape of the display screen are not particularly limited.

[0163] For example, if display component 24 includes a display panel, the display panel may use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLed, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. Furthermore, in a possible implementation, a touch panel in user interface 23 may be coupled to the display panel in display component 24. For example, the touch panel may be disposed below the display panel, and the touch panel may be configured to detect tactile pressure acting on the display panel when a user inputs a touch operation (e.g., a tap, a slide, or a touch) using the display panel, and the display panel is configured to display content.

[0164] The camera 25 may be a monocular camera, a twin-lens camera, or a depth camera and is configured to photograph / record the environment and acquire images / video images. The images / video images collected by the camera 25 may be used as input data for a SLAM system, for example, or the images / video may be displayed using the display component 24.

[0165] In a possible implementation, the camera 25 may be considered a sensor. The images collected by the camera 25 may be in IMG format or another format type, which is not specifically limited in the embodiments of the present application.

[0166] The sensor 26 may be configured to collect data related to status changes (e.g., rotation, shaking, movement, or vibration) of the electronic device 20. The data collected by the sensor 26 may be used as input data for the SLAM system. The sensor 26 may include one or more sensors, such as an inertial measurement unit (IMU) and a time-of-flight (TOF) sensor. The IMU may include sensors such as a gyroscope and an accelerometer. The gyroscope is configured to measure the angular velocity of the electronic device when the electronic device moves, and the accelerometer is configured to measure the acceleration of the electronic device when the electronic device moves. The TOF sensor may include a light emitter and a light receiver. The light emitter is configured to emit light, such as laser light, infrared light, or radar waves, outward. The light receiver is configured to detect reflected light, such as reflected laser light, infrared light, or radar waves.

[0167] It should be noted that the sensor 26 may further include many other sensors, such as an inertial sensor, a barometer, a magnetometer, and a wheel speedometer, etc. This is not specifically limited in the embodiments of the present application.

[0168] The positioning module 27 is configured to implement physical positioning of the electronic device 20, for example, to obtain an initial position of the electronic device 20. The positioning module 27 may include one or more of a Wi-Fi positioning module, a Bluetooth positioning module, a base station positioning module, and a satellite positioning module. To assist in the positioning, a global navigation satellite system (GNSS) may be disposed in the satellite positioning module. The GNSS includes, but is not limited to, the BeiDou system, the global positioning system (GPS) system, the global navigation satellite system (GLONASS) system, and the Galileo navigation satellite system (Galileo) system.

[0169] The transceiver 28 is configured to implement communication between the electronic device 20 and another device (e.g., a server or another electronic device). The transceiver 28 integrates a transmitter and a receiver configured to transmit and receive radio frequency signals, respectively. In a specific implementation, the transceiver 28 includes, but is not limited to, an antenna system, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec (CODEC) chip, a subscriber identity module (SIM) card, a storage medium, etc. In a possible implementation, the transceiver 28 may alternatively be implemented on a separate chip. The transceiver 28 supports at least one data network communication in at least one data network such as 2G / 3G / 4G / 5G, and / or supports at least one of the following short-range wireless communication methods: Bluetooth (BT) communication, wireless fidelity (Wi-Fi) communication, near field communication (NFC), infrared (IR) wireless communication, ultra wideband (UWB) communication, and ZigBee (ZigBee) protocol communication.

[0170] In this embodiment of the present application, the processor 211 operates program code stored in the memory 22 to perform various functional applications and data processing of the electronic device 20 .

[0171] Fig. 3 is a schematic diagram of the structure of a server 30 according to an embodiment of the present application. As shown in Fig. 3, the server 30 may be the server in the embodiment shown in Fig. 1. The server 30 includes a processor 301, a memory 302 (one or more computer-readable storage media), and a transceiver 303. These components may communicate with each other by using one or more buses 304.

[0172] The processor 301 may be one or more CPUs. If the processor 301 is one CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0173] The memory 302 may be connected to or coupled to the processor 301 via a bus 304 and configured to store various groups of program codes and / or instructions and data (e.g., map data and pose data). In a specific implementation, the memory 302 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), etc.

[0174] The transceiver 303 is mainly an integration of a receiver and a transmitter, where the receiver is configured to receive data (e.g., requests or images) sent by the electronic device, and the transmitter is configured to send data (e.g., map data or pose data) to the electronic device.

[0175] 3 is merely an example provided in this embodiment of the present application, and it should be understood that the server 30 may further include many more components than those shown in the figure, which is specifically not limited in this embodiment of the present application.

[0176] In this embodiment of the present application, the processor 301 runs program code stored in the memory 302 to perform various functional applications and data processing of the server 30 .

[0177] The term "coupled" as used in the present application means a direct connection or a connection through one or more intermediate components or circuits.

[0178] 4a is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in FIG. 4a, in this application scenario, an electronic device collects visual information by using a sensor, and determines the current pose of the electronic device by referring to the visual information from a server and a 3D map.

[0179] The 3D map is provided by a server. Specifically, the server generates a 3D map, then compresses the 3D map, and sends the compressed 3D map data to the electronic device. After receiving the compressed 3D map data, the electronic device performs decompression to obtain reconstructed data of the 3D map, and determines the current pose of the electronic device by referring to the collected visual information and the reconstructed 3D map data. The pose is position information of the electronic device, and may be an absolute pose in a world coordinate system or a pose relative to a point in the environment.

[0180] In this embodiment of the present application, the server may generate a 3D map in advance, compress the 3D map, and then store the compressed data of the 3D map locally. In this way, storage space can be saved. Furthermore, the server may send the compressed data of the 3D map to another device, such as cloud storage.

[0181] 1. The server generates a 3D map, compresses the 3D map to obtain compressed data of the 3D map, and stores the compressed data locally.

[0182] The server compresses the 3D map to save local storage space.

[0183] 2. The electronic device sends a map download request to the server. The map download request can be triggered in two ways.

[0184] (1) A user starts a map application installed on an electronic device, and the application uploads location information obtained based on GPS positioning or Wi-Fi positioning to a server corresponding to the application. The upload operation may trigger a map download request. Since the uploaded content includes location information, the server may perform a preliminary estimation based on the location information and send compressed data of a 3D map of the area to which the positioning point indicated by the location information belongs to the electronic device. The extent of the area to which the positioning point indicated by the location information belongs may be preset. For example, the area to which the positioning point belongs may be an administrative region of any level (including a county, city, country, or administrative region) in which the positioning point is located, or may be a circular area centered on the positioning point and using a specified distance as a radius.

[0185] (2) A user starts a map application installed on an electronic device and actively enters or selects an area on the application. For example, the user actively enters "xx business center" or selects "Street A" from a list of "Street A, Street B, and Street C." The user's aforementioned operation may trigger a map download request. Whether the user enters or selects a geographic location, the server accordingly transmits compressed data of a 3D map of the geographic location to the electronic device.

[0186] It should be understood that in this embodiment of the present application, in addition to the two methods described above, other methods may be used to trigger a map download request. For example, the electronic device automatically detects whether a condition for downloading a 3D map or starting a 3D map download is met, or the electronic device starts the 3D map download and requests the server to download a 3D map of an area range when it detects a change in ambient light or an environment change. The size of the area range is not particularly limited.

[0187] 3. The server sends the compressed data of the 3D map to the electronic device.

[0188] 4. Electronic devices collect visual information by using sensors.

[0189] It should be noted that steps 3 and 4 are independent of each other and the order in which they are performed is not limited.

[0190] 5. The electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.

[0191] 6. The electronic device performs positioning in a 3D map based on the visual information to obtain a pose corresponding to the visual information.

[0192] After receiving the compressed data of the 3D map, the electronic device does not need to immediately decompress the compressed data, but only needs to decompress the compressed data to obtain reconstructed data of the 3D map before performing positioning based on the visual information. For example, a user may pre-download compressed data of the 3D map of an area range by downloading an "offline map," and decompress the compressed data of the 3D map only when positioning is required.

[0193] 4b is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in FIG. 4b, in this application scenario, an electronic device collects visual information by using a sensor, and a server determines the current pose of the electronic device by referring to the visual information from the electronic device and a 3D map.

[0194] The 3D map is provided by a server. Specifically, the server generates the 3D map, then compresses the 3D map, and locally stores the compressed data of the 3D map. When receiving visual information from the electronic device, the server performs decompression to obtain reconstructed data of the 3D map, and determines the current pose of the electronic device by referring to the visual information and the reconstructed data of the 3D map.

[0195] 1. The server generates a 3D map, compresses the 3D map to obtain compressed data of the 3D map, and stores the compressed data locally.

[0196] 2. Electronic devices collect visual information by using sensors.

[0197] 3. The electronic device transmits the visual information to the server.

[0198] 4. The server decompresses the compressed data of the 3D map to obtain the reconstructed data of the 3D map.

[0199] It should be understood that the server compresses the 3D map to save storage space.

[0200] 5. The server performs positioning in the 3D map based on the visual information to obtain a pose corresponding to the visual information.

[0201] 6. The server sends the pause to the electronic device.

[0202] 4c is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in FIG. 4c, in this application scenario, an electronic device collects visual information by using a sensor, and determines a current pose of the electronic device by referring to the collected visual information and a 3D map.

[0203] The 3D map is provided by the electronic device. Specifically, the electronic device generates the 3D map, then compresses the 3D map, and locally stores the compressed data of the 3D map. When visual information is collected, the electronic device performs decompression to obtain reconstructed data of the 3D map, and determines the current pose of the electronic device by referring to the collected visual information and the reconstructed data of the 3D map.

[0204] 1. The electronic device generates a 3D map, compresses the 3D map to obtain compressed data of the 3D map, and locally stores the compressed data.

[0205] It should be understood that the electronic device may compress the 3D map to save storage space.

[0206] 2. Electronic devices collect visual information by using sensors.

[0207] 3. The electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.

[0208] 4. The electronic device performs positioning in a 3D map based on the visual information to obtain a pose corresponding to the visual information.

[0209] 4d is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in FIG. 4d, in this application scenario, the second electronic device collects visual information by using a sensor, and determines the current pose of the second electronic device by referring to the visual information and a 3D map from a server.

[0210] The 3D map is generated by the first electronic device. Specifically, the first electronic device generates the 3D map, compresses the 3D map, and then transmits the compressed 3D map data to a server. The server then transmits the compressed 3D map data to the second electronic device. The second electronic device performs decompression to obtain reconstructed data of the 3D map and determines the current pose of the second electronic device by referring to the collected visual information and the reconstructed data of the 3D map.

[0211] In this embodiment of the present application, the first electronic device may generate a 3D map in advance, compress the 3D map, and then send the compressed data of the 3D map to the server, thereby reducing the transmission bandwidth.

[0212] 1. A first electronic device generates a 3D map and compresses the 3D map to obtain compressed data of the 3D map.

[0213] 2. The first electronic device sends compressed data of the 3D map to the server.

[0214] The first electronic device compresses the 3D map and then transmits the compressed data of the 3D map, thereby reducing the transmission bandwidth and improving the transmission efficiency.

[0215] 3. The second electronic device sends a map download request to the server.

[0216] The second electronic device may send a map download request based on the trigger scheme shown in Figure 4a.

[0217] 4. The server sends the compressed data of the 3D map to the second electronic device.

[0218] 5. The second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.

[0219] 6. The second electronic device collects visual information by using a sensor.

[0220] 7. The second electronic device performs positioning in the 3D map based on the visual information and obtains a pose corresponding to the visual information.

[0221] 4d is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in FIG. 4d, in this application scenario, the second electronic device collects visual information by using a sensor, and determines the current pose of the second electronic device by referring to the visual information and a 3D map from a server.

[0222] The 3D map is generated by the first electronic device. Specifically, the first electronic device generates the 3D map, compresses the 3D map, and then transmits the compressed data of the 3D map to the server. The server performs decompression to obtain reconstructed data of the 3D map, and determines the current pose of the second electronic device by referring to the visual information from the second electronic device and the reconstructed data of the 3D map.

[0223] 1. A first electronic device generates a 3D map and compresses the 3D map to obtain compressed data of the 3D map.

[0224] 2. The first electronic device sends compressed data of the 3D map to the server.

[0225] 3. The second electronic device collects visual information by using a sensor.

[0226] 4. The second electronic device sends a positioning request to the server, where the positioning request carries the visual information.

[0227] 5. The server decompresses the compressed data of the 3D map to obtain the reconstructed data of the 3D map.

[0228] 6. The server performs positioning in the 3D map based on the visual information to obtain a pose corresponding to the visual information.

[0229] 7. The server transmits the pose obtained through positioning to the second electronic device.

[0230] Fig. 4f is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in Fig. 4f, in this application scenario, the second electronic device collects visual information by using a sensor, and determines the current pose of the second electronic device by referring to the visual information and the 3D map from the first electronic device.

[0231] The 3D map is generated by the first electronic device. Specifically, the first electronic device generates the 3D map, compresses the 3D map, and then transmits the compressed data of the 3D map to the second electronic device. The second electronic device performs decompression to obtain reconstructed data of the 3D map and determines the current pose of the second electronic device by referring to the collected visual information and the 3D map from the first electronic device.

[0232] 1. The first electronic device generates a 3D map, compresses the 3D map to obtain compressed data of the 3D map, and stores the compressed data locally.

[0233] 2. The second electronic device sends a map download request to the first electronic device.

[0234] 3. The first electronic device transmits the compressed data of the 3D map to the second electronic device.

[0235] 4. The second electronic device decompresses the compressed data of the 3D map to obtain reconstructed data of the 3D map.

[0236] 5. The second electronic device collects visual information by using a sensor.

[0237] 6. The second electronic device performs positioning in the 3D map based on the visual information to obtain a pose corresponding to the visual information.

[0238] The positioning algorithm used in the embodiment shown in Figures 4a-4f may include:

[0239] (1) From visual information, search An area descriptor of interest is extracted, where: search The algorithm used to extract area descriptors of an object is consistent with the algorithm for extracting area descriptors from a 3D map.

[0240] (2) From visual information, searchThe 3D map points of interest are extracted, search 3D map point space position of the object and search A 3D map point descriptor of the object is obtained, where: search The algorithm for extracting 3D map point descriptors of an object is consistent with the algorithm for extracting 3D map point descriptors from a 3D map.

[0241] (3) search Based on the area descriptor of interest, multiple area descriptors included in the 3D map data search is performed to obtain multiple candidate area descriptors.

[0242] In an embodiment of the present application, search A distance between the area descriptor of interest and each area descriptor in the plurality of area descriptors may be calculated. The distance may include Hamming distance, Manhattan distance, Euclidean distance, etc. Then, at least one area descriptor that satisfies a condition (e.g., the distance is less than a threshold) is selected as a candidate area descriptor.

[0243] (4) search Matching is performed separately between the target 3D map point descriptor and the 3D map point descriptors corresponding to the multiple candidate area descriptors. search The similarity between the target 3D map point descriptor and the 3D map point descriptors corresponding to the multiple candidate area descriptors is calculated separately to find the most similar 3D map point.

[0244] (5) The pose of the electronic device is calculated based on the discovered 3D map points by using pose solver algorithms such as perspective-n-point (PnP) camera pose estimation and efficient perspective-n-point camera pose estimation (EPnP).

[0245] In any one of the application scenarios in FIGS. 4a to 4f, positioning is performed based on the 3D map in the embodiment of the present application to obtain the current pose of the electronic device. The pose may be applied to fields such as AR navigation, AR human-computer interaction, driving assistance, and autonomous driving. For example, pose-based AR navigation is used as an example. FIG. 4g is a schematic diagram of a user interface displayed by an electronic device according to the embodiment of the present application. The electronic device may display the user interface shown in FIG. 4g based on the pose. The user interface may include a navigation arrow indication pointing toward the conference room 2, which may be a virtual object obtained from a server based on the pose or obtained locally based on the pose. The user interface may further include visual information collected by a sensor, such as the building shown in FIG. 4g. The user goes to the conference room 2 by referring to the user interface of the electronic device shown in FIG. 4g.

[0246] It should be noted that the 3D map reconstruction data in the embodiments of the present application obtained through unfolding may also be referred to as 3D map reconstruction data.

[0247] In the embodiment shown in Fig. 4a to Fig. 4f, compression and expansion are performed on the 3D map, and positioning is performed based on the reconstruction data of the 3D map obtained through expansion to obtain the current pose of the electronic device. Regarding positioning, in an implementable manner, positioning can be performed based on the reconstruction data of the 3D map obtained through full expansion. In another possible implementable manner, the 3D map provided in the following embodiments of the present application can be search for By using the method for the above, the descriptor can be obtained through screening, and the descriptor can be used to perform positioning. search for According to the method for search and decompression or extraction of compressed data alternately, resulting in multi-stage search The descriptor is obtained by performing the following: search In comparison, the 3D maps provided in the following embodiments of the present application are search for In the method for one stage non-complete expansion search At least two stages search By being in search Improve speed and use at least two stages search Another stage in search teeth search The accuracy can be guaranteed. Incomplete unfolding means that all of the reconstruction data of the 3D map is obtained through incomplete unfolding. In the following, we will refer to the 3D map as search for This section explains how to do this.

[0248] In the embodiment of the present application, the 3D map search means finding some most similar or most related 3D map points from multiple 3D map points on a 3D map.

[0249] The length of the binary data in the embodiment of the present application may be the number of bits of the binary data (also referred to as the number of bits). For example, if the binary data is 110, the number of bits in the binary data is 3.

[0250] Provided in the embodiments of the present application search The method can also be used in other technical fields, for example in the field of image or video data. search , audio data search , and point cloud data search Note that this can be applied to

[0251] Based on the above description, the 3D map according to the embodiment of the present application will be described below. search forProvided is a method therefor. For the sake of convenience, the embodiments of the method described below are expressed as a combination of a series of action steps. However, those skilled in the art should understand that the specific implementation of the technical solution of the present application is not limited to the order of the series of action steps described.

[0252] Please refer to FIG. 5A. FIG. 5A is a schematic flowchart of a segment of a method for search for a 3D map according to an embodiment of the present application. In some implementations, the method can be applied to an electronic device in any one of the examples of FIGS. 1 to 4f, or can be applied to a server in any one of the examples of FIGS. 1 to 4f. The method includes, but is not limited to, the following steps.

[0253] S101A: Extract binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map.

[0254] The binary data can be the first binary data, the second binary data,..., the nth binary data, or the like, where n is a positive integer. It should be understood that the "first", "second", and "third" in the first binary data, the second binary data, the third binary data, and the like are not in order, and are merely used to distinguish each other. Hereinafter, the nth binary data is used as an example for explanation.

[0255] The nth binary data of the S 3D map descriptors is extracted from the compressed data of the S 3D map descriptors.

[0256] The nth binary data of the S 3D map descriptors is extracted from the compressed data of the S 3D map descriptors, where S is a positive integer. S represents the number of 3D map descriptors that need to be search performed at stage i search S < T, where T represents the total number of 3D map descriptors in the 3D map.

[0257] The number of 3D map points corresponding to the S 3D map descriptors may be greater than or equal to S.

[0258] For example, the S 3D map descriptors may be S area descriptors or S 3D map point descriptors, where one area descriptor corresponds to multiple 3D map points, and one 3D map point descriptor corresponds to one 3D map point. For specific descriptions and descriptions thereof, please refer to the descriptions and descriptions in the preceding embodiments, and the details will not be described again in this specification.

[0259] The compressed data of the S 3D map descriptors may be obtained through compression of the S area descriptors or the S 3D map point descriptors. Compression in this specification may include compaction. Compaction may include binarization and quantization. The n-th binary data of the S 3D map descriptors may be partial data of the compressed data of each of the S 3D map descriptors. For example, the n-th binary data of the S 3D map descriptors may be partial or full binary data of the compressed data of each of the S 3D map descriptors.

[0260] The binarization may include, but is not limited to, iterative quantization (ITQ) hashing, locality-sensitive hashing (LSH), spectral hashing, or the like. For example, through binarization, the descriptor may be mapped into a Hamming space (also referred to as a binary space) to obtain binary data.

[0261] Quantization includes, but is not limited to, scalar quantization, vector quantization, product quantization, etc. For example, through quantization, the 3D map descriptor may be compressed into one or more quantization indexes, where the one or more quantization indexes are quantized data of the 3D map descriptor. Each quantization index in the one or more quantization indexes corresponds to one quantization center.

[0262] Optionally, the execution entity in this embodiment of the present application may perform entropy decoding on the compressed data of the S 3D map descriptors to extract the n-th binary data of the S 3D map descriptors from the compressed data.

[0263] S102A: search Based on the binary data of the descriptors, S 3D maps are generated for stage i. search to obtain P 3D map descriptors.

[0264] The n-th binary data is used as an example for explanation.

[0265] search Based on the n-th binary data of the descriptor, the n-th binary data of the S 3D map descriptors is search to obtain P 3D map descriptors.

[0266] search Descriptors are features corresponding to the real-world environment that are extracted from visual information collected by sensors in electronic devices.

[0267] In this specification search The descriptor is the search Target area descriptor or search It can be a 3D map point descriptor of the object. In other words, search Descriptors are extracted from visual information search may be an area descriptor of interest, search An area descriptor of an object describes some or all characteristics of multiple 3D map points. searchThe area descriptor of interest may be a vector, for example a 128-dimensional vector or a 512-dimensional vector. search The number of dimensions of the area descriptor of interest is not limited in the above example and may be other values. In this embodiment of the present application, examples are not described one by one. Alternatively, search Descriptors were extracted from visual information search may be a 3D map point descriptor of the object, search The 3D map point descriptor of an object is used to indicate the local features of the corresponding 3D map point. search The 3D map point descriptor of the object may be a vector, and the number of dimensions of the vector may be any positive integer.

[0268] search The manner of obtaining the descriptor may include, but is not limited to, the following manner: In a possible manner, the execution subject of this embodiment may obtain the descriptor sent by another device. search For example, this embodiment may be performed by the first electronic device shown in FIG. 1, and the first electronic device may receive a descriptor transmitted by the second electronic device. search In another example, this embodiment may be performed by the server shown in FIG. 1, which may receive a descriptor transmitted by the first electronic device or the second electronic device. search In another possible implementation, the execution subject of this embodiment may receive visual information transmitted by another device, search For example, this embodiment may be performed by a first electronic device shown in FIG. 1, where the first electronic device receives visual information transmitted by a second electronic device, and the first electronic device: search In another example, this embodiment may be performed by a server shown in FIG. 1, which may receive visual information transmitted by the first electronic device or the second electronic device, and the server may: searchIn yet another possible implementation manner, the execution entity of this embodiment detects a visual information collection operation input by a user, and in response to the operation, triggers a sensor of the execution entity to perform visual information collection on a real environment to acquire visual information; search For example, this embodiment may be performed by a first electronic device shown in FIG. 1 , in which when detecting a visual information collection operation input by a user, the first electronic device triggers a sensor of the first electronic device to perform visual information collection on a real environment to acquire visual information in response to the operation; search The descriptor is extracted from the visual information. The visual information collection operation may enable a sensor to collect visual information in a manner such as tapping, touching, sliding, or shaking, or may enable the sensor to collect visual information through voice control or in another manner. The embodiment of the present application is not limited thereto. For example, after the first electronic device detects a touch operation performed by a user on a capture button in a user interface, the camera of the first electronic device is started in response to the touch operation.

[0269] search The nth binary data of the descriptor is search The descriptor may be a partial or full binary data. search Perform binarization on the descriptors, search Get the binary data of the descriptor, search Part or all of the binary data of the descriptor search It may be selected (also called "extracted") as the nth binary data of the descriptor.

[0270] The execution body of this embodiment is: search In the n-th binary data of S 3D map descriptors, search Run search The P 3D map descriptors that are most similar or most relevant to the nth binary data of the descriptor may be obtained.

[0271] P is a positive integer, and P represents the number of 3D map descriptors obtained through screening at stage i, where P < T. search Here, P < T.

[0272] In some embodiments, P < S. For example, a subset of 3D map descriptors is obtained through stage i, and the number of 3D map descriptors in the subset of 3D map descriptors is smaller than the number of 3D map descriptors in the subset of 3D map descriptors that existed before stage i. In some embodiments, P may be greater than S. For a specific explanation and description thereof, refer to the following explanation and description of the embodiment shown in FIG. 7. Details will not be described again herein. search Here, the number of 3D map descriptors in the subset of 3D map descriptors is smaller than the number of 3D map descriptors in the subset of 3D map descriptors that existed before stage i. search For a specific explanation and description thereof, refer to the following explanation and description of the embodiment shown in FIG. 7. Details will not be described again herein.

[0273] When P < S, that is, when P 3D map descriptors belong to S 3D map descriptors, the similarity or correlation between the nth binary data of each of the P 3D map descriptors and the nth binary data of the descriptors is higher than that of the other 3D map descriptors in the S 3D map descriptors excluding the P 3D map descriptors. There can be many implementations for calculating the similarity or correlation. For example, calculate the first distance between the nth binary data of two 3D map descriptors (between the 3D map descriptor and the descriptor) to determine the correlation of the nth binary data of each of the two 3D map descriptors. A smaller first distance indicates a higher correlation. search Here, the similarity or correlation between the nth binary data of each of the P 3D map descriptors and the nth binary data of the descriptors is higher than that of the other 3D map descriptors in the S 3D map descriptors excluding the P 3D map descriptors. search There can be many implementations for calculating the similarity or correlation. For example, calculate the first distance between the nth binary data of two 3D map descriptors (between the 3D map descriptor and the descriptor) to determine the correlation of the nth binary data of each of the two 3D map descriptors. A smaller first distance indicates a higher correlation.

[0274] The first distance in the embodiments of the present application may include a distance obtained by using binary data such as a Hamming distance. For example, the first distance may be a Hamming distance. In another example, the first distance may be a distance obtained by obtaining the absolute value of the difference value between corresponding bits of two binary data and summing up the absolute values of all the difference values. Based on the first distance search means determining the correlation or similarity by calculating the first distance and screening the 3D map descriptors.

[0275] S103A: Perform stage m decompression on the compressed data of the P 3D map descriptors to obtain reconstructed data of the P 3D map descriptors.

[0276] The reconstructed data may be the first reconstructed data, the second reconstructed data, ..., the mth reconstructed data, or the like, where m is a positive integer. It should be understood that the terms "first," "second," and "third" in the first reconstructed data, the second reconstructed data, the third reconstructed data, and the like are not used in any order but merely to distinguish one from another. The following uses the mth reconstructed data as an example for explanation.

[0277] Stage m of decompression is performed on the compressed data of the P 3D map descriptors to obtain the m-th reconstructed data of the P 3D map descriptors.

[0278] In S101A and S102A, P 3D map descriptors can be obtained through screening, and in S103A, stage m expansion can be performed on the compressed data of the P 3D map descriptors to obtain m-th reconstructed data of the P 3D map descriptors, thereby search where m is a positive integer. The unfolding of stage m may include dequantization. The unfolding of stage m may be a staged unfolding (i.e., a portion of a full unfolding) or may be a full unfolding.

[0279] The reconstructed data of the P 3D map descriptors for stage m can be obtained by performing the unfolding of stage m on the partial data or all data of the compressed data of each of the P 3D map descriptors. For example, the reconstructed data of the P 3D map descriptors for stage m can be the dequantized data obtained by performing dequantization on a part or all of the compressed data of each of the P 3D map descriptors. It should be noted that the dequantized data can also be referred to as reconstructed data, and the reconstructed data can be obtained by the dequantization method in the prior art. In another example, dequantization is performed on the quantized data (a part of the compressed data) of each of the P 3D map descriptors to obtain P dequantized data. Then, the reconstructed data of the P 3D map descriptors for stage m can be obtained based on the P dequantized data of each of the P 3D map descriptors and the binary data (a part of the compressed data).

[0280] In some embodiments, the unfolding of stage m may further include other unfoldings such as entropy decoding and prediction. In the embodiments of the present application, examples are not described one by one.

[0281] S104A: search Execute at stage j in the reconstructed data of the P 3D map descriptors based on the partial data or all data of the descriptors search to obtain Q 3D map descriptors.

[0282] Q is a positive integer, 0 < Q < P, 0 < P < T, 0 < S ≤ T, where T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L is the search total number of stages of the 3D map or search the stage number threshold, and L is a positive integer greater than 1.

[0283] The reconstructed data may be the first reconstructed data, the second reconstructed data, ..., the mth reconstructed data, or the like, where m is a positive integer. It should be understood that the terms "first," "second," and "third" in the first reconstructed data, the second reconstructed data, the third reconstructed data, and the like are not used in any order but merely to distinguish one from another. The following uses the mth reconstructed data as an example for explanation.

[0284] search Partial or complete data of the descriptor is search It means some or all components of a descriptor. search The components of the descriptor are: search The value of one dimension of the descriptor vector. For example, search The descriptor may be a vector, for example an M-dimensional vector, where each dimension of the M dimensions is a component of the vector.

[0285] As in S102A, the execution subject of this embodiment is search In the m-th reconstruction data of P 3D map descriptors by using partial or full data of the descriptors, search Run search The Q most similar or most relevant 3D map descriptors to the m-th reconstruction data of the descriptor may be obtained, where Q may be less than P. search Another 3D map descriptor subset is obtained through, and the number of 3D map descriptors in this 3D map descriptor subset is search It can also be understood that the number of 3D map descriptors obtained through

[0286] the m-th reconstruction data for each of the Q 3D map descriptors, and search The similarity or correlation between the partial data or the whole data of the descriptor is higher than that of other 3D map descriptors in the P 3D map descriptors, except for the Q 3D map descriptors. There are many possible implementations for calculating the similarity or correlation. For example, the mth reconstruction data of two 3D map descriptors and the mth reconstruction data of the 3D map descriptor are searchA second distance (between partial data or all data of the descriptors) is calculated to determine the correlation of the two 3D map descriptors. The second distance may include, but is not limited to, Euclidean distance, dot product distance, cosine distance, Manhattan distance, and the like. For example, a smaller Euclidean distance indicates a higher correlation, and a larger dot product distance indicates a higher correlation.

[0287] Of course, it can be understood that a condition determination step can be performed after S104A. When the condition is met, search can terminate when the condition is not met. search The condition may be any condition and may be set appropriately as needed. For example, the condition may be that the similarity or correlation is greater than or equal to a preset threshold, or search In another example, the condition may be that the number of 3D map descriptors obtained through is less than or equal to a preset number, or the like. search The information may be obtained through

[0288] The 3D map in this embodiment of the present application is search for The method for is a segment of a procedure. Before and / or after the segment of a procedure, another stage of another stage may be performed. search In other words, the 3D map search for The method for this purpose may be a multi-stage method with more stages than those shown in FIG. 5A, for example, three stages, four stages, or five stages. search For example, more stages may be included before S101A in the embodiment shown in FIG. search or after S104A in the embodiment shown in FIG. 5A, search There are many possibilities. We won't go through each example here.

[0289] In some embodiments, a multi-stage searchThe Q 3D map descriptors obtained through can be used for positioning. The 3D map points corresponding to the N 3D map descriptors in the Q 3D map descriptors are search If the 3D map points corresponding to the N 3D map descriptors match, positioning may be performed based on the 3D map points corresponding to the N 3D map descriptors to obtain pose information of the electronic device, where the electronic device may be an electronic device that collects visual information, and <N≦Qである。

[0290] In this embodiment, binary data of S 3D map descriptors is extracted from the compressed data of S 3D map descriptors; search Based on the binary data of the descriptors, S 3D maps are generated for stage i. search Execute the step m to obtain P 3D map descriptors through screening; execute the step m to decompress the compressed data of the P 3D map descriptors to obtain P reconstructed data of the P 3D map descriptors; search In the reconstruction data of P 3D map descriptors based on partial or full data of the descriptors, search By performing screening, fewer 3D map descriptors are obtained. The binary data of the compressed data is search One stage in at least two stages of search Used for search The speed is improved and the reconstruction data obtained through the deployment is stored in a separate stage. search Used for search Accuracy is guaranteed. 3D map reconstruction data obtained through full unfolding search In comparison, the 3D map provided in this embodiment of the present application search for The way to do this is search Performance may be improved.

[0291] See Figure 5B, which illustrates a 3D map according to an embodiment of the present application. search for4 is a schematic flowchart of a method segment for: In some implementations, the method may be applied to any one of the electronic devices in the examples of Figures 1 to 4f, or may be applied to any one of the servers in the examples of Figures 1 to 4f. The method includes, but is not limited to, the following steps:

[0292] S101B: Extract first binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map.

[0293] The first binary data may be any one of the first binary data, the second binary data, ..., the nth binary data, or the like, where n is a positive integer. It should be understood that the "first," "second," and "third" in the first binary data, the second binary data, the third binary data, and the like are not used in any order but merely to distinguish one from another. In the following, the nth binary data will be used as an example for explanation.

[0294] The n-th binary data of the S 3D map descriptors is extracted from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points on the 3D map.

[0295] S102B: search In stage i, the first binary data of the S 3D map descriptors is calculated based on the first binary data of the descriptors. search to obtain P 3D map descriptors.

[0296] In the following, the n-th binary data is used as an example for explanation. search Based on the n-th binary data of the descriptor, the n-th binary data of the S 3D map descriptors is search to obtain P 3D map descriptors.

[0297] For a specific explanation and description of S101B and S102B, please refer to S101A and S102A in the embodiment shown in FIG. 5A.

[0298] S103B: Extract second binary data of the P number of 3D map descriptors from the compressed data of the P number of 3D map descriptors.

[0299] The second binary data may be any one of the second binary data, the third binary data, ..., the qth binary data, or the like, where q is a positive integer. It should be understood that the "second" and "third" in the second binary data, the third binary data, and the like are not used in order, but merely to distinguish each other. In the following, the qth binary data is used as an example for explanation. In the example, q=n+1.

[0300] P 3D map descriptors can be obtained from the S 3D map descriptors through screening in S101B and S102B, and then extracting the q-th binary data of the P 3D map descriptors from the compressed data of the P 3D map descriptors, and performing a calculation based on the q-th binary data. search The q-th binary data of the P 3D map descriptors may be partial or all of the compressed data of each of the P 3D map descriptors. For example, the q-th binary data of the P 3D map descriptors may be partial or all of the binary data of each of the P 3D map descriptors.

[0301] The n-th binary data of each of the S 3D map descriptors is different from the q-th binary data of each of the P 3D map descriptors. In an implementable manner, the difference may be a difference in length. Specifically, the length of the n-th binary data of each of the S 3D map descriptors is different from the length of the q-th binary data of each of the P 3D map descriptors. qthIt is different from the length of the binary data. In another implementable way, the difference can be the difference between the binarization methods corresponding to the n-th binary data and the q-th binary data. Specifically, the binarization method corresponding to the n-th binary data of each 3D map descriptor among the S 3D map descriptors is different from the binarization method corresponding to the q-th binary data of each 3D map descriptor among the P 3D map descriptors. For example, the n-th binary data of each 3D map descriptor among the S 3D map descriptors is obtained by a binarization method (e.g., locality-sensitive hashing), and each 3D map descriptor among the P 3D map descriptors is obtained by another binarization method (e.g., spectral hashing).

[0302] The length of the n-th binary data can be measured or expressed by using the number of bits in the n-th binary data. Similarly, the length of the q-th binary data can be measured or expressed by using the number of bits in the q-th binary data.

[0303] When the P 3D map descriptors belong to the S 3D map descriptors (P < S), the position of the n-th binary data of each 3D map descriptor among the P 3D map descriptors in the compressed data of the 3D map descriptors is different from the position of the q-th binary data of the 3D map descriptors in the compressed data of the 3D map descriptors. The position can be a start position, an end position, or the like.

[0304] In the example, the components of the original 3D map descriptor corresponding to the n-th binary data of each 3D map descriptor among the P 3D map descriptors are different from the components of the original 3D map descriptor corresponding to the q-th binary data of the corresponding 3D map descriptor. That is, the n-th binary data of the 3D map descriptor can be the binary data of the first part of the 3D map descriptor (e.g., the components in the first 128 dimensions of the 3D map descriptor), and the q-th binary data of the 3D map descriptor can be the binary data of the second part of the 3D map descriptor (e.g., the components in the last 128 dimensions of the 3D map descriptor).

[0305] When P 3D map descriptors belong to S 3D map descriptors (P < S), the length of the n-th binary data of each 3D map descriptor among the P 3D map descriptors may be smaller than the length of the q-th binary data of each 3D map descriptor among the P 3D map descriptors. The length of the n-th binary data of each 3D map descriptor in search is smaller, search improving the speed, and the length of the q-th binary data of the 3D map descriptor in search is larger in search to improve the accuracy. search The speed and search accuracy are comprehensively improved through the discrimination in at least two search stages. Alternatively, the length of the n-th binary data of each 3D map descriptor among the P 3D map descriptors may be larger than the length of the q-th binary data of each 3D map descriptor among the P 3D map descriptors. Alternatively, the length of the n-th binary data of each 3D map descriptor among the P 3D map descriptors may be equal to the length of the q-th binary data of each 3D map descriptor among the P 3D map descriptors.

[0306] S104B: search Execute search in the second binary data of the P 3D map descriptors based on the second binary data of the descriptors to obtain Q 3D map descriptors.

[0307] Hereinafter, the q-th binary data is used as an example for explanation.

[0308] search Execute search in the q-th binary data of the P 3D map descriptors based on the q-th binary data of the descriptors to obtain Q 3D map descriptors.

[0309] S, P, Q, i, and j are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, where T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L is the search Total number of stages or search represents the stage number threshold, where L is a positive integer greater than 1.

[0310] search The qth binary data of the descriptor is search The descriptor may be a partial or full binary data. search Perform binarization on the descriptors, search Get the binary data of the descriptor, search Part or all of the binary data of the descriptor search It may be selected (also called "extracted") as the (n+1)th representation of the descriptor. search The nth binary data of the descriptor is search Note that this may be the same as or different from the qth binary data of the descriptor.

[0311] As in S102B, the execution subject of this embodiment is search By using the qth binary data of the descriptor, search Run search The Q most similar or most relevant 3D map descriptors may be obtained for the qth binary data of the descriptor, where Q may be less than P. Another 3D map descriptor subset may be obtained for stage j. search The number of 3D map descriptors in this 3D map descriptor subset is obtained through stage i. search It can also be understood that the number of 3D map descriptors obtained through

[0312] the qth binary data of each of the Q 3D map descriptors, and search The similarity or correlation between the q-th binary data of the descriptor is higher than that of other 3D map descriptors in the P 3D map descriptors, excluding the Q 3D map descriptors. For the explanation and description of the manner of calculating the similarity or correlation, please refer to the relevant explanation and description in S102B. The details will not be described again in this specification.

[0313] Optionally, in S102B, search and in S104B, stage j search Used for search Both methods are based on distance first search For specific explanations and descriptions, please refer to the explanations and descriptions of S102A in the embodiment shown in Figure 5A. The details will not be described again in this specification.

[0314] Of course, it can be understood that after S104B, a condition determination step can be performed. When the condition is met, search can terminate when the condition is not met. search The condition may be any condition and may be set appropriately as needed. For example, the condition may be that the similarity or correlation is greater than or equal to a preset threshold, or search The number of 3D map descriptors obtained through may be less than or equal to a preset number, or the like. In this embodiment of the present application, examples are not described one by one.

[0315] The 3D map in this embodiment of the present application is search for The method for is a segment of a procedure. Before and / or after the segment of a procedure, another stage of another stage may be performed. search In other words, the 3D map search for The method for this purpose may be a multi-stage method with more stages than those shown in FIG. 5B, for example, three stages, four stages, or five stages. search For example, more stages may be included before S101B in the embodiment shown in FIG. search or after S104B in the embodiment shown in FIG. 5B, search There are many possibilities. We won't go through each example here.

[0316] In some embodiments, a multi-stage searchThe Q 3D map descriptors obtained through can be used for positioning. The 3D map points corresponding to the N 3D map descriptors in the Q 3D map descriptors are search If the 3D map points corresponding to the N 3D map descriptors match, positioning may be performed based on the 3D map points corresponding to the N 3D map descriptors to obtain pose information of the electronic device, where the electronic device may be an electronic device that collects visual information, and <N≦Qである。

[0317] In this embodiment, extracting first binary data of the S number of 3D map descriptors from the compressed data of the S number of 3D map descriptors; search Based on the first binary data of the descriptor, the first binary data of the S 3D map descriptors of stage i search Execute the following to obtain P 3D map descriptors through screening; extract second binary data of the P 3D map descriptors from the compressed data of the P 3D map descriptors; search Based on the second binary data of the descriptor, the second binary data of the P 3D map descriptors is calculated for stage j. search to obtain fewer 3D map descriptors through screening. search The binary data of the 3D map descriptor is extracted from the compressed data in one stage. search and alternating to perform at least two stages of 3D mapping in the compressed state. search , which results in search Get the 3D map descriptor for any two stages. search The information content held in the binary data of the 3D map descriptor used by each of the stages may be different, or the information content of any two stages may be different. search The information completeness of the binary data of the 3D map descriptor used by each of the methods is different. This results in the reconstructed data of the 3D map obtained through full unfolding. search When compared with search Improved speed, search Accuracy can be guaranteed.

[0318] Please refer to FIG. 5C. FIG. 5C is a schematic flowchart of a segment of a method for search for generating a 3D map according to an embodiment of the present application. In some implementations, the method can be applied to an electronic device in any one of the examples of FIGS. 1 to 4f, or can be applied to a server in any one of the examples of FIGS. 1 to 4f. The method includes the following steps, but is not limited thereto.

[0319] S101C: Perform the expansion of stage m on the compressed data of S 3D map descriptors to obtain the first reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map.

[0320] The first reconstructed data can be any one of the first reconstructed data, the second reconstructed data,..., the mth reconstructed data, or the like, where m is a positive integer. It should be understood that the "first", "second", and "third" in the first reconstructed data, the second reconstructed data, the third reconstructed data, and the like are not in order, but are simply used to distinguish each other. An example where the first reconstructed data is the mth reconstructed data is used as an example for explanation below.

[0321] Perform the expansion of stage m on the compressed data of S 3D map descriptors to obtain the mth reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map.

[0322] S and m are positive integers. S represents the number of 3D map descriptors that need to be search performed in stage i search . S < T, where T represents the total number of 3D map descriptors in the 3D map.

[0323] The number of a plurality of 3D map points corresponding to the S 3D map descriptors may be greater than S, or may be equal to S.

[0324] The compressed data of the S 3D map descriptors may be obtained through compression on the S area descriptors or the S 3D map point descriptors. Compression herein may include quantization. Optionally, compression may further include at least one of binarization, prediction, entropy, coding, or the like.

[0325] The m-th reconstruction data of the S 3D map descriptors is obtained by performing stage m decompression on the partial data of each of the compressed data of the S 3D map descriptors.

[0326] m is a positive integer. The expansion of stage m may include dequantization. The expansion of stage m may be a staged expansion (i.e., a portion of a full expansion).

[0327] The m-th reconstructed data of the S 3D map descriptors may be obtained by performing stage m decompression on partial data of the compressed data of each of the S 3D map descriptors. For example, the m-th reconstructed data of the S 3D map descriptors may be dequantized data obtained by performing dequantization on a portion of the compressed data of each of the S 3D map descriptors. Note that the dequantized data may also be referred to as reconstructed data, and the reconstructed data may be obtained by a dequantization method in the prior art. In another example, dequantization may be performed on a portion of the quantized data (a portion of the compressed data) of each of the S 3D map descriptors to obtain S dequantized data, and then the m-th reconstructed data of the S 3D map descriptors may be obtained based on the S dequantized data and a portion of the binary data (a portion of the compressed data) of each of the S 3D map descriptors.

[0328] In some embodiments, the development of stage m may further include other developments such as entropy decoding and prediction, etc. In this embodiment of the present application, examples will not be described one by one.

[0329] S102C: searchBased on the partial data or all data of the descriptor, perform the operation at stage i in the first reconstructed data of S 3D map descriptors to search obtain P 3D map descriptors.

[0330] An example where the first reconstructed data is the m-th reconstructed data is used as an example for explanation below.

[0331] search Based on the partial data or all data of the descriptor, perform the operation at stage i in the m-th reconstructed data of S 3D map descriptors to search obtain P 3D map descriptors.

[0332] search The partial data or all data of the descriptor search means some or all components of the descriptor. search The components of the descriptor search are the values of one dimension of the vector of the descriptor. For example, search the descriptor can be a vector, for example, an M-dimensional vector, where each dimension of the M dimensions is a component of the vector.

[0333] The execution entity of this embodiment search by using the partial data or all data of the descriptor, in the m-th reconstructed data of S 3D map descriptors search perform search to obtain P 3D map descriptors that are most similar or most relevant to the m-th reconstructed data of the descriptor.

[0334] In some embodiments, P < S. For example, a subset of 3D map descriptors is obtained through the operation at stage i, and the number of 3D map descriptors in the subset of 3D map descriptors search is the number of 3D map descriptors in the operation at stage i. searchis smaller than the number of 3D map descriptors in the 3D map descriptor subset that exists before it. In some embodiments, P may be greater than S. For a specific explanation and description thereof, refer to the following explanation and description of the embodiment shown in FIG. 7. Details will not be described again herein.

[0335] P < S, that is, when P 3D map descriptors belong to S 3D map descriptors, the similarity or correlation between the m-th reconstructed data of each of the P 3D map descriptors and search the partial data or all data of the descriptors is higher than that of the other 3D map descriptors in the S 3D map descriptors excluding the P 3D map descriptors. There can be many implementations for calculating the similarity or correlation. For example, calculate the second distance (between the k-th reconstructed data of the 3D map descriptors and search the partial data or all data of the descriptors) of two 3D map descriptors to determine the correlation of the two 3D map descriptors. The second distance can include, but is not limited to, Euclidean distance, inner product distance, cosine distance, Manhattan distance, and the like. For example, a smaller Euclidean distance indicates a higher correlation, and a larger inner product distance indicates a higher correlation.

[0336] S103C: Execute the expansion of stage k on the compressed data of the P 3D map descriptors to obtain the second reconstructed data of the P 3D map descriptors, where the expansion process of the compressed data of the S 3D map descriptors includes the expansion of stage m and the expansion of stage k, or the expansion process of the compressed data of the S 3D map descriptors includes the expansion of stage k, and the expansion of stage k includes the expansion of stage m.

[0337] The second reconstructed data can be any one of the second reconstructed data, the third reconstructed data, …, the k-th reconstructed data, or the like, where k is a positive integer and m < k. It should be understood that the “second” and “third” in the second reconstructed data, the third reconstructed data, and the like are not in order but are merely used to distinguish each other. An example where the second reconstructed data is the k-th reconstructed data is used as an example for explanation below.

[0338] Execute the expansion of stage k on the compressed data of P 3D map descriptors to obtain the k-th reconstructed data of the P 3D map descriptors, where the expansion process of the compressed data of S 3D map descriptors includes the expansion of stage m and the expansion of stage k, or the expansion process of the compressed data of S 3D map descriptors includes the expansion of stage k, and the expansion of stage k includes the expansion of stage m.

[0339] In S101C and S102C, P 3D map descriptors can be obtained by screening from S 3D map descriptors, and then, execute the expansion of stage k on the compressed data of the P 3D map descriptors to obtain the k-th reconstructed data of the P 3D map descriptors, whereby search is further executed. In some embodiments, k = m + 1.

[0340] The k-th reconstructed data of the P 3D map descriptors can be obtained by executing the expansion of stage k on the partial data or all data of the compressed data of each of the P 3D map descriptors.

[0341] In some embodiments, the search of stage i and the search of stage j search Both methods used are based on the second distance. searchThe method includes: P 3D map descriptors belonging to S 3D map descriptors; and the distortion of the m-th reconstructed data of each 3D map descriptor of the P 3D map descriptors is different from the distortion of the k-th reconstructed data of each 3D map descriptor of the P 3D map descriptors; the distortion of the m-th reconstructed data of each 3D map descriptor of the P 3D map descriptors indicates a degree of difference between the m-th reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor; and the distortion of the k-th reconstructed data of each 3D map descriptor of the P 3D map descriptors indicates a degree of difference between the k-th reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor, where P <Sである。

[0342] In some embodiments, the distortion of the m-th reconstructed data of each 3D map descriptor of the P 3D map descriptors is greater than the distortion of the k-th reconstructed data of each 3D map descriptor of the P 3D map descriptors. search At least two stages of the next stage search The results are: search is smaller than the distortion of the previous stage, search Improve the accuracy of the results.

[0343] For example, P 3D map descriptors belong to S 3D map descriptors. One of the P 3D map descriptors is used as an example, where m=1 and k=2. The compressed data of the 3D map descriptor includes first quantized data and second quantized data. The first reconstructed data of the 3D map descriptor is obtained by performing stage 1 decompression on the first quantized data, and the first reconstructed data of the 3D map descriptor may include reconstructed data of some components of the original vector of the 3D map descriptor. The second reconstructed data of the 3D map descriptor is obtained by performing stage 2 decompression on the second quantized data, and the second reconstructed data of the 3D map descriptor may include reconstructed data of some other components of the original vector of the 3D map descriptor. That is, the decompression process of the compressed data of the 3D map descriptor includes stage 1 (m=1) decompression and stage 2 (k=2) decompression.

[0344] For example, P 3D map descriptors belong to S 3D map descriptors. One of the P 3D map descriptors is used as an example, where m=1 and k=2. The compressed data of the 3D map descriptor includes first quantized data and second quantized data. Unlike the previous example, the second quantized data in this specification may be obtained after quantization is performed on the residual data of the first quantized data, and the first reconstructed data of the 3D map descriptor is obtained by performing stage 1 unfolding on the first quantized data, and the first reconstructed data of the 3D map descriptor may include the first reconstructed data (low-precision reconstructed data) of the original vector of the 3D map descriptor. The second reconstructed data of the 3D map descriptor is obtained by performing stage 2 unfolding on the second quantized data, and the second reconstructed data of the 3D map descriptor may include the reconstructed data of the residual data of the 3D map descriptor. The first reconstructed data of the 3D map descriptor and the second reconstructed data of the 3D map descriptor (high-precision reconstructed data) are summed to obtain the reconstructed data of the original vector of the 3D map descriptor. That is, the decompression process of the compressed data of the 3D map descriptor includes stage 2 (k=2) decompression, which in turn includes stage 1 (m=1) decompression.

[0345] S104C: search In the second reconstruction data of P 3D map descriptors based on partial data or all data of the descriptors, search to obtain Q 3D map descriptors.

[0346] An example in which the second reconstructed data is the k-th reconstructed data is used below as an example for explanation.

[0347] search Based on partial or full data of the descriptors, the k-th reconstruction data of the P 3D map descriptors is search to obtain Q 3D map descriptors.

[0348] S, P, Q, i, j, k, and m are positive integers, where 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in a 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the total number of search stages or search a stage number threshold, and L is a positive integer greater than 1.

[0349] For a specific description and illustration of S104C, refer to the description and illustration of S104A in the embodiment shown in FIG. 5A. Details will not be described again herein.

[0350] Of course, it can be understood that a condition determination step can be executed after S104C. When the condition is satisfied, in this embodiment search can end. When the condition is not satisfied, one stage or multi-stage search can continue to be executed. The condition can be any condition and can be appropriately set as needed. For example, the condition can be that the similarity or correlation is greater than or equal to a preset threshold, or search the number of 3D map descriptors obtained through

[0351] The method for search for the 3D map in this embodiment of the present application is a segment of a procedure. Before and / or after the segment of the procedure, another stage of another stage in another stage search can be further included. In other words, the method for search for the 3D map can involve more stages than those shown in FIG. 5C, for example, a multi-stage search including 3 stages, 4 stages, or 5 stages. For example, before S101C in the embodiment shown in FIG. 5C, there can be more stages of search or after S104C in the embodiment shown in FIG. 5C, there can be more stages of searchThere are many possibilities. We won't go through each example here.

[0352] In some embodiments, a multi-stage search The Q 3D map descriptors obtained through can be used for positioning. The 3D map points corresponding to the N 3D map descriptors in the Q 3D map descriptors are search If the 3D map points corresponding to the N 3D map descriptors match, positioning may be performed based on the 3D map points corresponding to the N 3D map descriptors to obtain pose information of the electronic device, where the electronic device may be an electronic device that collects visual information, and <N≦Qである。

[0353] In this embodiment, performing stage m decompression on the compressed data of the S 3D map descriptors to obtain first reconstructed data of the S 3D map descriptors; search In the first reconstruction data of S 3D map descriptors based on partial data or all data of the descriptors, search to obtain P 3D map descriptors through screening; and to perform stage k expansion on the compressed data of the P 3D map descriptors to obtain second reconstructed data of the P 3D map descriptors; search In the second reconstruction data of P 3D map descriptors based on partial data or all data of the descriptors, search By performing screening, fewer 3D map descriptors are obtained. search Obtaining the reconstruction data of the 3D map descriptor is a first stage. search At least two stages are executed alternately. search In one stage search is a staged development, search Improve speed. At least two stages search The differentiated deployment settings for search Accuracy can be guaranteed. searchThe reconstruction data of the 3D map descriptor used by the method has different degrees of unfolding or distortion. This allows the reconstruction data of the 3D map obtained through full unfolding to have different degrees of unfolding or distortion. search When compared with search Improved speed, search Accuracy can be guaranteed.

[0354] See Figure 5D, which illustrates a 3D map according to an embodiment of the present application. search for 4 is a schematic flowchart of a method segment for: In some implementations, the method may be applied to any one of the electronic devices in the examples of Figures 1 to 4f, or may be applied to any one of the servers in the examples of Figures 1 to 4f. The method includes, but is not limited to, the following steps:

[0355] S101D: Perform stage m of decompression on the compressed data of S 3D map descriptors to obtain reconstructed data of S 3D map descriptors, where the S 3D map descriptors correspond to multiple 3D map points of the 3D map.

[0356] The reconstructed data may be the first reconstructed data, the second reconstructed data, ..., the mth reconstructed data, or the like, where m is any one of positive integers. It should be understood that the "first," "second," and "third" in the first reconstructed data, the second reconstructed data, the third reconstructed data, and the like are not used in any order but merely to distinguish one from another. An example in which the reconstructed data is the mth reconstructed data will be used below as an example for explanation.

[0357] Perform stage m of decompression on the compressed data of the S 3D map descriptors to obtain reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map.

[0358] S102D: search In the reconstruction data of S 3D map descriptors based on partial or full data of the descriptors, searchExecute to obtain P 3D map descriptors.

[0359] For the descriptions and explanations of S101D and S102D, please refer to the descriptions and explanations of S101C and S102C in FIG. 5C. Details will not be explained again in this specification.

[0360] S103D: Extract the binary data of P 3D map descriptors from the compressed data of P 3D map descriptors.

[0361] S104D: search Based on the binary data of the descriptors, perform search at stage j in the binary data of P 3D map descriptors to obtain Q 3D map descriptors.

[0362] S, P, Q, i, m, and j are positive integers, where 0 < Q < P, 0 < P < T, 0 < S ≤ T. T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents the search total number of stages or search stage number threshold of the 3D map, and L is a positive integer greater than 1.

[0363] For the descriptions and explanations of S103D and S104D, please refer to the descriptions and explanations of S103B and S104B in FIG. 5B. Details will not be explained again in this specification.

[0364] In this embodiment, perform the expansion at stage m on the compressed data of S 3D map descriptors to obtain the reconstructed data of S 3D map descriptors, search Based on the partial data or all data of the descriptors, perform search at stage i in the reconstructed data of S 3D map descriptors to obtain P 3D map descriptors through screening. Extract the binary data of P descriptors from the compressed data of P 3D map descriptors, search Based on the binary data of the descriptors, perform at stage j in the binary data of P 3D map descriptors search 3D map descriptors are extracted from compressed data and extracted through screening. search By alternating between these, we can create a multi-stage search to obtain a 3D map descriptor that can be used for positioning. search One stage in search is the expansion, search Improve accuracy and use at least two stages search One stage in search is the extraction from the compressed data, search Improve the speed of 3D map reconstruction data obtained through full unfolding search In comparison, the 3D map provided in this embodiment of the present application search for The way to do this is search It can improve speed, search Accuracy can be guaranteed.

[0365] As shown in Figures 5A to 5C search Method steps can be flexibly combined to create a variety of different multi-stage search Note that the method can be formed.

[0366] Please refer to Figure 6. Figure 6 shows a 3D map according to an embodiment of the present application. search for 4 is a schematic flowchart of a method for implementing the present invention. In some implementations, the method may be applied to the electronic device in any one of the examples of FIGS. 1 to 4f, or may be applied to the server in any one of the examples of FIGS. 1 to 4f. In this embodiment, a multi-stage search 4 stages search For illustration purposes, we will use the example of Stage 1. search and Stage 2 search is based on the first distance search Using the method, Stage 3 search and Stage 4 search is based on the second distance search The method includes, but is not limited to, the following steps:

[0367] The following method steps in this embodiment of the present application begin to be performed.

[0368] S201: search Get the descriptor. search Descriptors are features corresponding to the real-world environment that are extracted from visual information collected by sensors in electronic devices.

[0369] For the explanation and description of S201, please refer to the explanation and description of S102A in the embodiment shown in Figure 5A. The details will not be described again in this specification.

[0370] S202: Extracting first (n=1) binary data of S1 3D map descriptors from the compressed data of S1 3D map descriptors; search Get the first (n=1) binary data of the descriptor.

[0371] The S1 3D map descriptors can be S1 area descriptors or S1 3D map point descriptors. One area descriptor corresponds to multiple 3D map points. For specific explanations and descriptions, please refer to the explanations and descriptions in the preceding embodiments. The details will not be described again in this specification.

[0372] For the explanation and description of the compressed data of the S1 3D map descriptors, please refer to the embodiment shown in Figures 5A to 5D, and the details will not be described again in this specification.

[0373] search The first binary data of the descriptor is search In the example, the execution subject of this embodiment is search Perform binarization on the descriptor search Get the binary data of the descriptor, search Part of the binary data of the descriptor search It may be selected as the first binary data of the descriptor.

[0374] S203: search Stage 1 (i=1) for the first binary data of S1 3D map descriptors based on the first (n=1) binary data of the descriptor search to obtain P1 3D map descriptors.

[0375] The execution body of this embodiment is: search By using the first binary data of the descriptor, the first distance-based search Run search The subset may be obtained by obtaining P1 3D map descriptors that are most similar or most relevant to the first binary data of the descriptor, where P1 is less than S1. search The number of 3D map descriptors in the subset is obtained through search It can also be understood that the number of 3D map descriptors in the 3D map descriptor subset that is present before the first 3D map descriptor is smaller than the number of 3D map descriptors in the 3D map descriptor subset that is present before the first 3D map descriptor.

[0376] S204: Extract second (n=2) binary data of the P1 3D map descriptors from the compressed data of the P1 3D map descriptors; search Get the second (n=2) binary data of the descriptor.

[0377] search The second (n=2) binary data of the descriptor is search The descriptor may be partial or full binary data. search The number of bits in the second (n=2) binary data of the descriptor is search This is different from the number of bits in the first (n=1) binary data of the descriptor. For example, search The number of bits in the second (n=2) binary data of the descriptor is search The number of bits in the first (n=1) binary data of the descriptor is larger than the number of bits in the first (n=1) binary data of the descriptor. In this embodiment, the bit overhead of the second binary data may be higher than the bit overhead of the first binary data. search of search The speed can be improved and the next stage search The accuracy of the can be guaranteed.

[0378] S205: search Stage 2 (j=2) based on the second (n=2) binary data of the descriptors P1 3D map descriptors search to obtain Q1 3D map descriptors.

[0379] The execution body of this embodiment is: search The first distance-based feature is obtained by using the second binary data of the P1 3D map descriptors. search Run search Q1 3D map descriptors that are most similar or most relevant to the second binary data of the descriptor may be obtained, where Q1 is less than or equal to P1. Another subset may be search The number of 3D map descriptors in this subset is obtained through search It can also be understood that the number of 3D map descriptors in the 3D map descriptor subset obtained through

[0380] S206: Perform stage 1 (m=1) decompression on the compressed data of the Q1 3D map descriptors to obtain first (m=1) reconstructed data of the Q1 3D map descriptors.

[0381] S207: search The first (m=1) reconstruction data of Q1 3D map descriptors based on partial or full data of the descriptors is used in stage 3. search to obtain Q2 3D map descriptors.

[0382] In this example, the execution body of this embodiment is: search By using partial or full data of the descriptors, the second distance base is calculated in the first (m=1) reconstruction data of the Q1 descriptors. search Run searchQ2 3D map descriptors that are most similar or most relevant to the partial or full data of the descriptor may be obtained, where Q2 is less than or equal to Q1. Another subset may be search The number of 3D map descriptors in this subset is obtained through search It can also be understood that the number of 3D map descriptors in the 3D map descriptor subset obtained through

[0383] Based on the first distance search Compared to the previous two stages, search is the second distance base search Using search Improve accuracy.

[0384] S208: Perform stage 2 (k=2) unfolding on the Q2 3D map descriptors to obtain second (k=2) reconstruction data of the Q2 3D map descriptors.

[0385] In the example, the distortion of the first (m=1) reconstructed data and the second (k=2) reconstructed data of the 3D map descriptor are different. In this embodiment, the distortion of the first (m=1) reconstructed data of the 3D map descriptor is higher than the distortion of the second (k=2) reconstructed data. Thus, search The accuracy of the can be guaranteed.

[0386] S209: search Stage 4: The second (k=2) reconstruction of Q2 3D map descriptors based on the entire descriptor data. search to obtain Q3 3D map descriptors.

[0387] In the example, the execution body of this embodiment is: search By using all the data of the descriptors, in the second (k=2) reconstruction data of the Q2 3D map descriptors, search Run searchThe Q3 3D map descriptors that are most similar or most relevant to the total data of the descriptor may be obtained, where Q3 is less than or equal to Q2. Another subset may be search The number of 3D map descriptors in this subset is obtained through search It can also be understood that the number of 3D map descriptors in the 3D map descriptor subset obtained through

[0388] S210: Perform positioning based on the 3D map points corresponding to the Q3 3D map descriptors to obtain pose information of the electronic device.

[0389] In this embodiment, the binary data of the 3D map descriptor is extracted from the compressed data, or search The compressed data is decompressed to obtain the reconstructed data of the 3D map descriptor, and the multi-stage reconstruction of the 3D map is performed. search 3D map descriptors that can be used for positioning are obtained. search When compared with search Improved speed, search Accuracy can be guaranteed.

[0390] Please refer to Figure 7. Figure 7 shows a 3D map according to an embodiment of the present application. search for 5A is a schematic flowchart of a method for generating a 3D map descriptor for a computer system according to the present invention. In some implementations, the method may be applied to the electronic device in any one of the examples in FIGS. 1 to 4f, or may be applied to the server in any one of the examples in FIGS. 1 to 4f. This embodiment is described by using an example in which the S 3D map descriptors in the embodiment shown in FIG. 5A are S representative 3D map descriptors. Each of the S representative 3D map descriptors corresponds to at least one dataset. A specific implementation of 102A in FIG. 5A may include, but is not limited to, the following steps.

[0391] S1021: searchBased on the binary data of the descriptors, S representative 3D maps are generated for stage i. search to obtain at least one representative 3D map descriptor.

[0392] At least one representative 3D map descriptor corresponds to at least one dataset, each of the at least one dataset including compressed data of a plurality of 3D map descriptors, and the 3D map descriptors in each dataset have a correlation or similarity.

[0393] For the binary data of S representative 3D map descriptors, search The binary data of the descriptor is search The descriptor may be a partial or full binary data. search The first distance-based algorithm is used to find the binary data of the S representative 3D map descriptors. search Run search At least one representative 3D map descriptor that is most similar or most relevant to the binary data of the descriptor may be obtained, and the number of the at least one representative 3D map descriptor is less than or equal to S.

[0394] S1022: 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor are used as P 3D map descriptors.

[0395] search At least one representative 3D map descriptor is obtained through the at least one dataset, thereby obtaining compressed data of the P 3D map descriptors in the at least one dataset.

[0396] For example, S representative 3D map descriptors may be T1, T2, and T3, where T1 corresponds to dataset 1, T2 corresponds to dataset 2, and T3 corresponds to dataset 3. searchto obtain at least one representative 3D map descriptor T3 through the above, to obtain a dataset 3, where the dataset 3 may include compressed data of the P 3D map descriptors.

[0397] In this embodiment, one stage of the 3D map descriptor is used for the less representative 3D map descriptor. search Run search Next, we perform one-stage or multi-stage algorithms on datasets corresponding to representative 3D map descriptors. search Run search This can improve the accuracy of the reconstruction data of the 3D map obtained through full unfolding. search When compared with search Improved speed, search Accuracy can be guaranteed.

[0398] It should be noted that the specific implementation of the S 3D map descriptors in the embodiment shown in Figures 5C to 5D as S representative 3D map descriptors is similar to that of the embodiment shown in Figure 7. For the specific implementation of 102B in Figure 5B, 102C in Figure 5C, or 102D in Figure 5D, please refer to the embodiment shown in Figure 7. The details will not be described again in this specification.

[0399] Please refer to Figures 8 and 9. Figure 8 shows a 3D map according to an embodiment of the present application. search for FIG. 9 is a schematic flow chart of a method for generating a 3D map according to an embodiment of the present application. search for 1 is a schematic diagram of a processing process of a method for performing a network access control process according to an embodiment of the present invention. The embodiment includes a server and an electronic device. The method includes, but is not limited to, the following steps:

[0400] S401: The server determines, based on each component of the S 3D map descriptors and the corresponding component of the preset threshold vector, a magnitude relationship between each component of the S 3D map descriptors and the corresponding component of the preset threshold vector.

[0401] S402: The server performs binarization on the magnitude relationship to obtain binary data of each of the S 3D map descriptors.

[0402] S403: The server performs quantization on the absolute value of the difference value between each component of the S 3D map descriptors and the corresponding component of the preset threshold vector to obtain quantized data of each of the S 3D map descriptors.

[0403] Each component included in the preset threshold vector is an arbitrary value.

[0404] S404: The server encapsulates the binary data and quantized data of each of the S 3D map descriptors to obtain a bitstream of the 3D map.

[0405] S405: The server transmits a bitstream of the 3D map to the electronic device.

[0406] S406: Electronic devices search Get the descriptor.

[0407] Referring to FIG. 9, the electronic device may generate a visual image from the visual information collected by the sensor. search Descriptors may be extracted.

[0408] S407: The electronic device decapsulates the 3D map bitstream to obtain binary data and quantized data for each of the S 3D map descriptors, and extracts the binary data for each of the S 3D map descriptors from the binary data and quantized data for each of the S 3D map descriptors.

[0409] 9, the S 3D map descriptors are used as an example to be five 3D map descriptors (T1, T2, T3, T4, and T5), and the electronic device may decapsulate the 3D map bitstream to obtain the binary data and quantized data of each of the five 3D map descriptors shown in FIG.

[0410] S408: The electronic device search Based on the binary data of the descriptor, the binary data of each of the S 3D map descriptors is calculated in stage 1. search to obtain P 3D map descriptors.

[0411] search The binary data of the descriptor is obtained in the following manner: search Based on each component of the descriptor and the corresponding component of the preset threshold vector, search A magnitude relationship between each component of the descriptor and the corresponding component of the preset threshold vector may be determined. A binarization may be performed on the magnitude relationship to search Obtain the binary data of the descriptors. Stage 1 is performed on the binary data of each of the S 3D map descriptors. search is the first distance base search The method may be used.

[0412] Figure 9 is used as an example for further explanation. search teeth, search The five 3D map descriptors (T1, T2, T3, T4, and T5) are processed based on the binary data of the descriptors. For example, the binary data of each of the five 3D map descriptors (T1, T2, T3, T4, and T5) and search The Hamming distance between the binary data of the descriptors can be calculated, and the 3D map descriptor is obtained through screening based on the Hamming distance. In this example, two 3D map descriptors are obtained through screening, that is, T1 and T2 in the dotted frame shown in Figure 9 are obtained through screening. That is, P=2.

[0413] S409: The electronic device unpacks the quantized data of each of the P 3D map descriptors to obtain reconstructed data of each of the P 3D map descriptors.

[0414] The electronic device may perform dequantization on the quantized data of each of the P 3D map descriptors to obtain dequantized data for each of the P 3D map descriptors. Reconstructed data for each of the P 3D map descriptors is obtained based on the dequantized data and the binary data of each of the P 3D map descriptors.

[0415] S410: An electronic device search In stage 2, the reconstruction data for each of the P 3D map descriptors is search to obtain Q 3D map descriptors.

[0416] Stage 2 is performed on the reconstructed data for each of the P 3D map descriptors. search is the second distance base search The method may be used.

[0417] Figure 9 is used as an example for further explanation. search teeth, search It is performed in T1 and T2 based on all components of the descriptor. For example, the reconstruction data of each of T1 and T2 and search The Euclidean distance between the descriptors can be calculated, and the 3D map descriptor is obtained through screening based on the Euclidean distance. In this example, one 3D map descriptor is obtained through screening, that is, T1 shown in Figure 9 is obtained through screening. That is, Q=1.

[0418] In this embodiment, the server compresses the S 3D map descriptors in the 3D map to obtain binary data and quantized data for each of the S 3D map descriptors, thereby reducing the resource overhead required to transmit the 3D map. The electronic device obtains the binary data for the S 3D map descriptors; search The first distance-based method is based on the binary data of the descriptor and the binary data of the S 3D map descriptors. search P 3D map descriptors are quickly obtained through screening in searchImprove speed. The electronic device obtains reconstruction data for P 3D map descriptors, search A second distance-based method based on the reconstruction data of the descriptor and P 3D map descriptors search Q 3D map descriptors are obtained through accurate screening in this method. search Improve accuracy.

[0419] The above describes the 3D map in the embodiment of the present application with reference to the accompanying drawings. search for The method for generating a 3D map in the embodiment of the present application will be described in detail below with reference to FIGS. search for We describe a device for creating a 3D map. search for In the embodiment of the present application, the apparatus for search for It should be understood that the method for the 3D map in the embodiment of the present application can be implemented as follows. search for When describing an apparatus for the above, repetitive descriptions will be omitted below as appropriate.

[0420] Please refer to Figure 10. Figure 10 shows a 3D map according to an embodiment of the present application. search for 10 is a schematic diagram of the structure of the device for generating a 3D map. search for The apparatus 1000 for search It may include a module 1001, an extraction module 1002, and an unpacking module 1003.

[0421] In a first possible implementation, the extraction module 1002 is configured to extract binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map. search Module 1001 is search Based on the binary data of the descriptor, the binary data of S 3D map descriptors in stage i search to obtain P 3D map descriptors, where: searchA descriptor is a feature extracted from visual information collected by sensors of an electronic device that corresponds to a real environment. The expansion module 1003 is configured to perform the expansion of stage m on the compressed data of P 3D map descriptors to obtain the reconstructed data of the P 3D map descriptors. Here, the expansion process of the compressed data of the P 3D map descriptors includes at least the expansion of stage m. search Module 1001 search Based on partial data or all data of the descriptor, perform search at stage j in the reconstructed data of the P 3D map descriptors to obtain Q 3D map descriptors. Here, S, P, Q, i, j, and m are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents search the total number of stages of the 3D map or search the stage number threshold, and L is a positive integer greater than 1.

[0422] In some embodiments, N 3D map descriptors among the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors. N is a positive integer, 0 < N ≤ Q, and the device further includes a pose determination module. The pose determination module performs positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0423] In some embodiments, the search used for stage i search method is a method based on the first distance search and the method used for stage j search is a method based on the second distance search and is a method based on the second distance. search

[0424] In some embodiments, the device further: search receives a descriptor, search ​Execute binarization on the descriptor to search obtain the binary data of the descriptor; or receive visual information and extract search the descriptor from the visual information, search execute binarization on the descriptor to search obtain the binary data of the descriptor; or trigger the sensor to collect visual information for the real environment to execute visual information collection in response to the visual information collection operation input by the user, and obtain the visual information, search extract the descriptor from the visual information, search execute binarization on the descriptor to search obtain the binary data of the descriptor, and is configured with an acquisition module.

[0425] In some embodiments, when N < Q, the expansion module 1003 further executes the expansion of stage k on the compressed data of Q 3D map descriptors to obtain the reconstructed data of Q 3D map descriptors; and, search execute search at stage r in the reconstructed data of Q 3D map descriptors based on the partial data or all data of the descriptor to obtain N 3D map descriptors, where N < Q < P, and the expansion process of the compressed data of P 3D map descriptors includes the expansion of stage m and the expansion of stage k, where r and k are positive integers, m < k, j < r ≤ L.

[0426] In some embodiments, the compressed data of the P 3D map descriptors includes binary data and quantized data of each of the P 3D map descriptors, and the decompression module 1003 is specifically configured to: perform dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data, where the P dequantized data are used as reconstruction data for the P 3D map descriptors; perform dequantization on the quantized data of each of the Q 3D map descriptors to obtain Q dequantized data; and obtain reconstruction data for each of the Q 3D map descriptors based on the Q dequantized data and the binary data of each of the Q 3D map descriptors.

[0427] In some embodiments, when N=Q, the compressed data of the P 3D map descriptors includes binary data and quantized data of each of the P 3D map descriptors; the decompression module 1003 is specifically configured to: perform dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data; and obtain reconstructed data of each of the P 3D map descriptors based on the P dequantized data and the binary data of each of the P 3D map descriptors.

[0428] In some embodiments, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset comprising at least one 3D map descriptor; search Module 1001: search Based on the binary data of the descriptor, S representative 3D maps are generated for stage i in the binary data of the descriptor. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0429] In a second possible implementation, the extraction module 1002 is configured to extract the first binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map. search Module 1001 search Based on the first binary data of the descriptors, in the first binary data of the S 3D map descriptors, at stage i search is configured to execute to obtain P 3D map descriptors, where search the descriptors are features extracted from visual information collected by sensors of an electronic device corresponding to the real environment. The extraction module 1002 is further configured to extract the second binary data of the P 3D map descriptors from the compressed data of the P 3D map descriptors. search Module 1001 search Based on the second binary data of the descriptors, in the second binary data of the P 3D map descriptors, at stage j search is configured to execute to obtain Q 3D map descriptors, where S, P, Q, i, and j are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L is the search total number of stages or search stage number threshold of the 3D map, and L is a positive integer greater than 1.

[0430] In some embodiments, N 3D map descriptors among the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors, N is a positive integer, 0 < N ≤ Q, and the device further includes a pose determination module. The pose determination module is configured to execute positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0431] In some embodiments, at stage i search and stage j search Both used in search The method is based on the first distance search The P 3D map descriptors belong to the S 3D map descriptors, and the position of the first binary data of each of the P 3D map descriptors in the compressed data of the 3D map descriptors is different from the position of the second binary data of the 3D map descriptor in the compressed data of the 3D map descriptors, and P <Sである。

[0432] In some embodiments, the length of the first binary data of each 3D map descriptor of the P 3D map descriptors is less than the length of the second binary data of each 3D map descriptor of the P 3D map descriptors.

[0433] In some embodiments, the device further comprises: search Receive the descriptor, search Perform binarization on the descriptors, search Obtaining the first binary data and the second binary data of the descriptor; or receiving the visual information and extracting the first binary data and the second binary data from the visual information. search Extract the descriptors, search Perform binarization on the descriptors, search acquiring first binary data and second binary data of the descriptor; or triggering a sensor to perform visual information gathering on a real environment in response to a visual information gathering operation input by a user to acquire visual information; search Descriptors are extracted from visual information, search Perform binarization on the descriptor search An acquisition module configured to acquire the first binary data and the second binary data of the descriptor is included.

[0434] In some embodiments, search the length of the first binary data of the descriptor is equal to the length of the first binary data of each 3D map descriptor of the S 3D map descriptors; and / or search The length of the second binary data of the descriptor is equal to the length of the second binary data of each 3D map descriptor of the S 3D map descriptors.

[0435] In some embodiments, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset comprising at least one 3D map descriptor; search Specifically, module 1001 includes: search In stage i, the first binary data of the S representative 3D map descriptors is calculated based on the first binary data of the descriptors. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0436] In a third possible implementation, the unfolding module 1003 is configured to perform stage m unfolding on the compressed data of the S 3D map descriptors to obtain first reconstruction data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map. search Module 1001 is search Based on the partial data or the entire data of the descriptors, the first reconstruction data of the S 3D map descriptors is search to obtain P 3D map descriptors, where: search The descriptors are features corresponding to the real environment extracted from visual information collected by a sensor of the electronic device. The unfolding module 1003 is further configured to perform a stage k unfolding on the compressed data of the P 3D map descriptors to obtain second reconstruction data of the P 3D map descriptors, where the unfolding process of the compressed data of the S 3D map descriptors includes a stage m unfolding and a stage k unfolding, or the unfolding process of the compressed data of the S 3D map descriptors includes a stage k unfolding and the stage k unfolding includes a stage m unfolding. search Module 1001 is searchBased on some or all of the descriptors, in the second reconstructed data of P 3D map descriptors at stage j search configured to execute to obtain Q 3D map descriptors, where S, P, Q, i, j, k, and m are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L is the search total number of stages of the 3D map or search represents the stage number threshold, and L is a positive integer greater than 1.

[0437] In some embodiments, N 3D map descriptors among the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors, N is a positive integer, 0 < N ≤ Q, and the apparatus further includes a pose determination module. The pose determination module is configured to perform positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0438] In some embodiments, both the search used in stage i and the search used in stage j search methods are methods based on the second distance, the P 3D map descriptors belong to S 3D map descriptors, and the distortion of the first reconstructed data of each 3D map descriptor of the P 3D map descriptors is different from the distortion of the second reconstructed data of each 3D map descriptor of the P 3D map descriptors. The distortion of the first reconstructed data of each 3D map descriptor of the P 3D map descriptors indicates the degree of difference between the first reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor, and the distortion of the second reconstructed data of each 3D map descriptor of the P 3D map descriptors indicates the degree of difference between the second reconstructed data of each 3D map descriptor and the corresponding original 3D map descriptor, where P < S.

[0439] ​​In some embodiments, the distortion of the first reconstructed data for each 3D map descriptor of the P 3D map descriptors is greater than the distortion of the second reconstructed data for each 3D map descriptor of the P 3D map descriptors.

[0440] In some embodiments, the device further comprises: search Receive the descriptor, search Acquire partial or full data of the descriptor; or receive visual information; search Descriptors are extracted from visual information, search Acquire partial or full data of the descriptor; or trigger a sensor to perform visual information acquisition on the real environment in response to a visual information acquisition operation input by a user to acquire visual information; search Extracting descriptors from visual information search It includes an acquisition module configured to acquire partial or full data of the descriptor.

[0441] In some embodiments, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset comprising at least one 3D map descriptor; search Specifically, module 1001 includes: search Based on the partial or full data of the descriptors, the m-th reconstruction data of the S representative 3D map descriptors is obtained in the stage i. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0442] In a fourth possible implementation, the decompression module 1003 is configured to perform stage m decompression on the compressed data of the S 3D map descriptors to obtain reconstructed data of the S 3D map descriptors, where the S 3D map descriptors correspond to a plurality of 3D map points of the 3D map. searchModule 1001 is configured to search perform at stage i in the reconstructed data of S 3D map descriptors based on partial or all data of the descriptors, search so as to obtain P 3D map descriptors, where search the descriptors are features corresponding to the real environment, extracted from the visual information collected by the sensors of the electronic device. The extraction module 1002 is configured to extract the binary data of P descriptors from the compressed data of P 3D map descriptors. search Module 1001 is further configured to search perform at stage j in the binary data of P 3D map descriptors based on the binary data of the descriptors, search so as to obtain Q 3D map descriptors, where S, P, Q, i, m, and j are positive integers, 0 < Q < P, 0 < P < T, 0 < S ≤ T, T represents the total number of 3D map descriptors in the 3D map, j = i + 1, 1 ≤ i < L, 1 < j ≤ L, and L represents search the total number of stages of the 3D map or search the stage number threshold, and L is a positive integer greater than 1.

[0443] In some embodiments, N 3D map descriptors among the Q 3D map descriptors are used for positioning, and the 3D map points corresponding to the N 3D map descriptors search coincide with the 3D map points corresponding to the descriptors. N is a positive integer, 0 < N ≤ Q, and the device further includes a pose determination module. The pose determination module is configured to perform positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain the pose information of the electronic device.

[0444] In some embodiments, the search method used for search stage i is a search method based on the first distance, and the search method used for search stage j is a search method based on the second distance.

[0445] In some embodiments, the device further comprises: search Receive the descriptor, search Perform binarization on the descriptors, search Obtain the binary data of the descriptor; or receive the visual information and search Extract the descriptors, search Perform binarization on the descriptors, search Acquiring binary data of the descriptor; or triggering a sensor to perform visual information acquisition on the real environment in response to a visual information acquisition operation input by a user to acquire visual information; search Descriptors are extracted from visual information, search Perform binarization on the descriptor search An acquisition module configured to acquire binary data of the descriptor is provided.

[0446] In some embodiments, the S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponding to at least one dataset, each dataset of the at least one dataset comprising at least one 3D map descriptor; search Specifically, module 1001 includes: search Based on the partial or full data of the descriptors, the m-th reconstruction data of the S representative 3D map descriptors is obtained in the stage i. search to obtain at least one representative 3D map descriptor; and using 3D map descriptors in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

[0447] 3D Map search for The apparatus 1000 for generating a 3D map in the embodiment shown in any one of FIGS. 5A to 5D or 6 or 7 is search forIt should be noted that the present invention may implement the method for the above-described embodiment, or may implement the relevant contents of S406 to S410 in the embodiment shown in Fig. 8, or may implement the relevant contents of the embodiment shown in Fig. 9. For the principles and technical effects of specific implementations, please refer to the specific descriptions of the above-described embodiment of the method. The details will not be described again in this specification.

[0448] 11 is a schematic block diagram of an implementation of a decoding device 1100 according to an embodiment of the present application. The decoding device 1100 may include a processor 1101, a memory 1102, and a bus system 1103. The processor 1101 and the memory 1102 are connected through the bus system 1103. The memory 1102 is configured to store instructions. The processor 1101 executes the instructions stored in the memory 1102 to generate the 3D map described herein. search for To avoid repetition, the details will not be described again here.

[0449] In this embodiment of the present application, the processor 1101 may be a central processing unit (CPU), or the processor 1101 may be another general-purpose processor, a DSP, an ASIC, an FPGA, or another programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or the like. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0450] The memory 1102 may include a ROM device or a RAM device. Any other suitable type of storage device may also be used as the memory 1102. The memory 1102 may include code and data 11021 that is accessed by the processor 1101 through the bus 1103. The memory 1102 may further include an operating system 11023 and applications 11022. The applications 11022 may include a processor for generating the 3D maps described herein. search forFor example, application 11022 may include applications 1-N, which may be used to generate 3D maps as described herein. search for The method may further include a 3D map application that performs the method for:

[0451] The bus system 1103 may further include a power bus, a control bus, a status signal bus, and the like in addition to a data bus. However, for clarity of explanation, various types of buses are illustrated in the figures as the bus system 1103.

[0452] Optionally, the decoding device 1100 may further include one or more output devices, such as a display 1104. In an example, the display 1104 may be a touch display that combines a display and a touch unit that operatively detects touch input. The display 1104 may be connected to the processor 1101 via a bus 1103.

[0453] The decoding device 1100 is a device for decoding a 3D map in the present application. search for It should be noted that the method for

[0454] Those skilled in the art will appreciate that the functions described with reference to the various illustrative logical blocks, modules, and algorithm steps disclosed and described herein can be implemented by hardware, software, firmware, or any combination thereof. If these functions are implemented in software, the functions described with reference to the illustrative logical blocks, modules, and steps may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or any communication medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be a medium that allows one or more computers or one or more processors to transmit instructions, code, and / or data structures for implementing the techniques described herein. search for The computer program product may include computer-readable media.

[0455] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other compact disk storage, magnetic disk storage, flash memory, or any other computer-accessible medium that can be used to store desired program code in the form of instructions or data structures. Additionally, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or another remote source over coaxial cable, fiber optic, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio waves, or microwave, the coaxial cable, fiber optic, twisted pair, DSL, or wireless technologies such as infrared, radio waves, or microwave are included within the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, and indeed refer to non-transitory, tangible storage media. As used herein, "disk" and "disc" include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), and Blu-ray discs. Disks typically replicate data magnetically, while discs replicate data optically by using lasers. Combinations of the above items should also be included within the scope of computer-readable media.

[0456] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor," as used herein, may refer to the aforementioned structure or any other structure that may be adapted to implement the techniques described herein. Furthermore, in some aspects, the functionality described in connection with the exemplary logic blocks, modules, and steps described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into an integrated codec. Furthermore, these techniques may be implemented entirely in one or more circuits or logic elements.

[0457] The techniques of the present application may be implemented in a variety of apparatuses or devices, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to highlight functional aspects of apparatus configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. In practice, as noted above, the various units may be combined into a codec hardware unit in conjunction with appropriate software and / or firmware, or may be provided by interoperating hardware units (including one or more processors as described above).

[0458] The above description is merely a specific implementation of the present application, and the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application shall be included in the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

Claims

1. 1. A method for searching a 3D map, the method comprising: extracting binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, wherein the compressed data of the S 3D map descriptors includes binary data and quantized data for each of the S 3D map descriptors, and the S 3D map descriptors correspond to a plurality of 3D map points of a 3D map; performing a stage i search in the binary data of the S 3D map descriptors based on binary data of a search descriptor to obtain P 3D map descriptors, where the search descriptors correspond to real-world environments and are features extracted from visual information collected by a sensor of an electronic device; performing stage m decompression on the compressed data of the P 3D map descriptors to obtain reconstructed data of the P 3D map descriptors, wherein the decompression process of the compressed data of the P 3D map descriptors includes at least the stage m decompression; and performing a search at stage j in the reconstructed data of the P 3D map descriptors based on partial or full data of the search descriptors to obtain Q 3D map descriptors; wherein S, P, Q, i, j, and m are positive integers, 0<Q<P, 0<P<T, 0<S≦T, T represents the total number of 3D map descriptors in the 3D map, j=i+1, 1≦i<L, 1<j≦L, L represents the total number of search stages or a search stage number threshold for the 3D map, and L is a positive integer greater than 1; method.

2. N 3D map descriptors among the Q 3D map descriptors are used for positioning, and 3D map points corresponding to the N 3D map descriptors match 3D map points corresponding to the search descriptor, where N is a positive integer and 0<N≦Q; the method further comprises: performing positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain pose information of the electronic device; The method of claim 1 , comprising:

3. The method according to claim 1 or 2, wherein the search method used for searching the stage i is a search method based on a first distance, and the search method used for searching the stage j is a search method based on a second distance.

4. The method further comprises: receiving the search descriptor and performing binarization on the search descriptor to obtain the binary data of the search descriptor; receiving the visual information, extracting the search descriptor from the visual information, and performing binarization on the search descriptor to obtain the binary data of the search descriptor; or In response to a visual information gathering operation input by a user, triggering the sensor to perform gathering of the visual information with respect to the real environment to acquire the visual information, extracting the search descriptor from the visual information, and performing binarization on the search descriptor to acquire the binary data of the search descriptor. The method of any one of claims 1 to 3, comprising:

5. When N<Q, the method further comprises: performing stage k decompression on the compressed data of the Q 3D map descriptors to obtain reconstructed data of the Q 3D map descriptors; and performing a stage r search in the reconstructed data of the Q 3D map descriptors based on partial or full data of the search descriptors to obtain the N 3D map descriptors; Equipped with 3. The method of claim 2, wherein N<Q<P, and the decompression process of the compressed data of the P 3D map descriptors includes decompressing the stage m and decompressing the stage k, where r and k are positive integers, m<k, and j<r≦L.

6. the compressed data for the P 3D map descriptors includes binary data and quantized data for each of the P 3D map descriptors; The step of performing stage m decompression on the compressed data of the P number of 3D map descriptors to obtain reconstructed data of the P number of 3D map descriptors; performing dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data, wherein the P dequantized data are used as the reconstruction data for the P 3D map descriptors. Including; The step of performing stage k decompression on the compressed data of the Q number of 3D map descriptors to obtain reconstructed data of the Q number of 3D map descriptors comprises: performing dequantization on the quantized data of each of the Q 3D map descriptors to obtain Q dequantized data; and obtaining reconstructed data for each of the Q 3D map descriptors based on the Q dequantized data and the binary data for each of the Q 3D map descriptors; The method of claim 5 , comprising:

7. When N=Q, the compressed data of the P 3D map descriptors includes binary data and quantized data of each of the P 3D map descriptors, and the step of performing stage m decompression on the compressed data of the P 3D map descriptors to obtain reconstructed data of the P 3D map descriptors comprises: performing dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data; and obtaining reconstructed data for each of the P 3D map descriptors based on the P dequantized data and the binary data for each of the P 3D map descriptors. The method of claim 2 , comprising:

8. The S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one dataset, and each dataset of the at least one dataset includes at least one 3D map descriptor, and the step of performing a search in the binary data of the S 3D map descriptors based on binary data of a search descriptor to obtain P 3D map descriptors includes: performing a search of the stage i in the binary data of the S representative 3D map descriptors based on the binary data of the search descriptor to obtain at least one representative 3D map descriptor; and using a 3D map descriptor in the data set corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

8. The method of claim 1, comprising:

9. 1. An apparatus for searching a 3D map, the apparatus comprising: an extraction module configured to extract binary data of S 3D map descriptors from the compressed data of the S 3D map descriptors, where the compressed data of the S 3D map descriptors includes binary data and quantized data for each of the S 3D map descriptors, and the S 3D map descriptors correspond to a plurality of 3D map points of a 3D map; a search module configured to perform a stage i search in the binary data of the S 3D map descriptors based on binary data of a search descriptor to obtain P 3D map descriptors, where the search descriptors include features corresponding to a real environment extracted from visual information collected by a sensor of an electronic device; and a decompression module configured to perform stage m decompression on the compressed data of the P 3D map descriptors to obtain reconstructed data of the P 3D map descriptors, wherein the decompression process of the compressed data of the P 3D map descriptors includes at least the stage m decompression; wherein: the search module is configured to perform a search of stage j in the reconstructed data of the P 3D map descriptors based on partial data or all data of the search descriptors to obtain Q 3D map descriptors, where S, P, Q, i, j, and m are positive integers, 0<Q<P, 0<P<T, 0<S≦T, T represents a total number of 3D map descriptors in the 3D map, j=i+1, 1≦i<L, 1<j≦L, L represents a total number of search stages or a search stage number threshold of the 3D map, and L is a positive integer greater than 1; Device.

10. N 3D map descriptors among the Q 3D map descriptors are used for positioning, and 3D map points corresponding to the N 3D map descriptors are consistent with 3D map points corresponding to the search descriptor, where N is a positive integer and 0<N≦Q; the apparatus further comprises a pose determination module, wherein: the pose determination module performs positioning based on the 3D map points corresponding to the N 3D map descriptors to obtain pose information of the electronic device.

10. The apparatus of claim 9.

11. 11. The apparatus according to claim 9, wherein the search method used for searching the stage i is a first distance-based search method, and the search method used for searching the stage j is a second distance-based search method.

12. The apparatus further comprises: receiving the search descriptor and performing binarization on the search descriptor to obtain the binary data of the search descriptor; receiving the visual information, extracting the search descriptor from the visual information, and performing binarization on the search descriptor to obtain the binary data of the search descriptor; or In response to a visual information gathering operation input by a user, trigger the sensor to perform gathering of the visual information with respect to the real environment to acquire the visual information, extract the search descriptor from the visual information, and perform binarization on the search descriptor to acquire the binary data of the search descriptor. The acquisition module is configured as follows:

12. The apparatus of claim 9, comprising:

13. When N<Q, the unfolding module further: performing stage k decompression on the compressed data of the Q 3D map descriptors to obtain reconstructed data of the Q 3D map descriptors; and performing a stage r search in the reconstructed data of the Q 3D map descriptors based on partial or full data of the search descriptors to obtain the N 3D map descriptors; It is structured as follows:

11. The apparatus of claim 10, wherein N<Q<P, and the decompression process of the compressed data of the P 3D map descriptors includes decompressing the stage m and decompressing the stage k, where r and k are positive integers, m<k, and j<r≦L.

14. The compressed data of the P 3D map descriptors includes binary data and quantized data for each of the P 3D map descriptors, and the decompression module specifically: performing dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data, where the P dequantized data are used as the reconstruction data for the P 3D map descriptors; performing dequantization on the quantized data of each of the Q 3D map descriptors to obtain Q dequantized data; and and obtaining reconstructed data for each of the Q 3D map descriptors based on the Q dequantized data and the binary data for each of the Q 3D map descriptors.

14. The device of claim 13, configured to:

15. When N=Q, the compressed data of the P 3D map descriptors includes binary data and quantized data of each of the P 3D map descriptors; the decompression module specifically: performing dequantization on the quantized data of each of the P 3D map descriptors to obtain P dequantized data; and obtaining reconstructed data for each of the P 3D map descriptors based on the P dequantized data and the binary data for each of the P 3D map descriptors; The device of claim 10, configured to:

16. The S 3D map descriptors are S representative 3D map descriptors, each of the S representative 3D map descriptors corresponds to at least one dataset, each dataset of the at least one dataset including at least one 3D map descriptor, and the search module: Based on the binary data of the search descriptor, perform a stage i search in the binary data of the S representative 3D map descriptors to obtain at least one representative 3D map descriptor; and and using a 3D map descriptor in the dataset corresponding to each of the at least one representative 3D map descriptor as the P 3D map descriptors.

16. The device according to any one of claims 9 to 15, configured to:

17. one or more processors; and a memory configured to store one or more programs; wherein: When the one or more programs are executed by the one or more processors, the one or more processors are capable of implementing the method of any one of claims 1 to 8. A device for searching 3D maps.

18. A computer program, which when run on a computer, enables the computer to carry out the method according to any one of claims 1 to 8.

Images