Terminal posture determination method and related device

Abstract

The present application provides a terminal posture determination method and related devices, the method comprising: obtaining multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature, the simulated building outline feature is obtained based on satellite images; obtaining building outline features in pictures taken by the terminal; and determining the simulated posture information corresponding to the simulated building outline feature that matches the building outline feature among the multiple simulated building outline features as the terminal's posture information. The implementation of the present application can achieve low-cost and high-precision acquisition of the terminal's 6DOF information, solving the problem of lack of accurate position information and posture information in application service requirements such as AR / VR.

Description

Technical Field

[0001] The present application relates to position and posture (posture) measurement technology, and in particular to a terminal posture determination method and related devices. Background Art

[0002] With the gradual maturity of the high-bandwidth and low-latency features of the fifth-generation (5G) mobile communication technology, as well as the development of mobile phone camera hardware and computing power, intelligent application services based on visual augmented reality (AR) / virtual reality (VR) are becoming more and more abundant.

[0003] AR / VR technology provides application services that are easy for users to interact and understand by superimposing virtual objects and real space. Taking AR navigation applications as an example, in AR indoor and outdoor positioning and navigation service scenarios, virtual navigation icons are superimposed on real scenes through cameras, and the current location information and navigation routes are fed back to users in the most intuitive way. At the same time, with the gradual maturity of artificial intelligence technology, deep learning has made major breakthroughs in the field of image understanding, such as object recognition, semantic segmentation, target tracking, etc., which has promoted the development of AR / VR intelligent application services. Using deep learning to intelligently identify target objects in real scenes can further enhance the intelligent application of AR / VR, such as identifying whether there are people, vehicles or obstacles in front and giving response plans, identifying houses and trees and assisting in providing navigation information, etc. Compared with traditional two-dimensional map navigation that uses abstract features of points, lines and surfaces to reflect the surrounding map environment, AR navigation positioning simulates the world seen by human eyes, combined with artificial intelligence to recognize semantic information, it can provide richer and more intuitive information, and is not easy to cause misunderstandings among users.

[0004] In order to overlay virtual objects on the correct position of the terminal screen, AR technology usually requires both the position information and the attitude information of the camera. However, traditional satellite positioning methods (such as Global Positioning System (GPS) positioning, GLONASS positioning, Galileo satellite navigation system positioning, Beidou positioning and other satellite positioning technologies) can only provide position information, but not attitude information.

[0005] Existing terminals (such as smart phones, etc.) usually have built-in sensors such as gyroscopes and magnetometers, which can estimate the terminal's posture information so that virtual objects can be projected to the correct position on the terminal screen. Combined with satellite positioning methods to provide terminal location information, the purpose of simultaneously obtaining location information and posture information on smartphones is achieved to meet the basic conditions for providing AR / VR services. Position information is usually represented by coordinates X, Y, and Z in Euclidean space, and posture information is represented by rotating coordinates pitch angle (pitch), yaw angle (yaw), and roll angle (roll). Therefore, position information (X, Y, Z) and posture information (yaw, pitch, roll) can be collectively referred to as 6-degree-of-freedom position and posture (6DOF) information.

[0006] However, the above methods usually cannot meet the needs of AR / VR services. On the one hand, the location information of the terminal obtained by the satellite positioning method is relatively rough, and the accuracy is usually in the range of several meters to tens of meters; on the other hand, the measurement accuracy of the built-in inertial sensors (such as gyroscopes and magnetometers) of existing terminals (such as smart phones, etc.) is generally low, and the attitude estimation error is large. For example, the gyroscope integrates the angular velocity per second to obtain the angle offset relative to the initial moment, which is prone to cumulative errors when running for a long time; the magnetometer measures the angle with the north direction to obtain angle information, but it is easily affected by the magnetic field in the environment and produces violent fluctuations.

[0007] Therefore, how to simultaneously obtain accurate 6DOF information on the terminal becomes a technical problem that needs to be solved urgently. Summary of the invention

[0008] The present application provides a terminal posture determination method and related devices, which can achieve low-cost and high-precision acquisition of the terminal's 6DOF information, and solve the problem of lack of accurate position information and posture information in application service requirements such as AR / VR.

[0009] In a first aspect, an embodiment of the present application provides a method for determining a terminal posture, the method comprising: obtaining a plurality of simulated building outline features and simulated posture information corresponding to each simulated building outline feature, wherein the simulated building outline feature is obtained based on a 3D model based on a satellite image; obtaining building outline features in a picture taken by a terminal; and determining the simulated posture information corresponding to a simulated building outline feature that matches the building outline feature among the plurality of simulated building outline features as the posture information of the terminal.

[0010] It can be seen that the embodiment of the present application mainly realizes low-cost and high-precision acquisition of 6DOF information of the terminal from two aspects:

[0011] On the one hand, the data source of this application is a 3D model based on satellite images. Based on the 3D model, effective information is extracted to construct a building outline feature library for 6DOF positioning. Compared with traditional posture acquisition methods (such as the posture detection hardware equipment provided by the mobile phone, large-scale professional posture measurement equipment to build a point cloud feature library, etc.), the embodiment of this application provides a low-cost data source.

[0012] On the other hand, in order to solve the problem that the bird's-eye view of satellite images and the head-on view of terminal images are difficult to correctly match, this application extracts the building contours in the image and matches them with the building contour feature library to obtain complete and high-precision position and angle information, thereby providing complete position information and posture information (i.e. 6DOF information) for subsequent AR / VR application services, thereby improving the user experience of application services.

[0013] Based on the first aspect, in a possible embodiment, before determining the simulated posture information corresponding to the simulated building outline feature that matches the building outline feature among the multiple simulated building outline features as the posture information of the terminal, it also includes: matching the building outline feature with the multiple simulated building outline features to obtain the simulated building outline feature that matches the building outline feature among the multiple simulated building outline features.

[0014] For example, the simulated building outline feature of the to-be-matched point with the highest matching degree may be used as the simulated building outline feature that matches the building outline feature.

[0015] Based on the first aspect, in a possible embodiment, before matching the building outline feature with the multiple simulated building outline features, it also includes: obtaining initial position information and initial posture information of the terminal, and the initial position information is obtained by collecting sensors in the terminal; accordingly, matching the building outline feature with the multiple simulated building outline features to obtain simulated building outline features that match the building outline feature among the multiple simulated building outline features includes: selecting a portion of simulated building outline features from multiple simulated building outline features according to the initial position information and initial posture information of the terminal; matching the building outline feature with the portion of simulated building outline features; and using the simulated building outline feature with the highest degree of matching among the portion of simulated building outline features as the simulated building outline feature that matches the building outline feature.

[0016] It can be seen that the embodiment of the present application makes full use of the sensors in the existing terminal (such as positioning module, gyroscope and magnetometer) to obtain the initial position information and posture information, uses the camera to collect pictures and extract building outline features, and uses the initial position information and posture information to determine a small part of the simulated building outline features for matching, that is, to build a retrieval buffer, and use the retrieval buffer to reduce the search range, speed up the retrieval efficiency, and achieve robust retrieval and similarity matching. By jointly calculating the similarity of position and posture, the position and posture of the sampling point with the highest similarity are used as the position and posture of the terminal, and 6DOF positioning is achieved on the terminal.

[0017] Based on the first aspect, in a possible embodiment, the matching of the building outline feature with the portion of simulated building outline features includes: calculating the initial position information and initial posture information of the building outline feature, and the similarity between the model position information and simulated posture information corresponding to each simulated building outline feature in the portion of simulated building outline features; the simulated building outline feature with the greatest similarity is used as the simulated building outline feature with the highest degree of matching.

[0018] In one example, the embodiment of the present application may use Euclidean distance as a similarity criterion in similarity matching to measure the similarity between the building feature line of the image and each building outline feature code in the retrieval buffer, and take the position and angle with the highest similarity as the posture information of the terminal's position information. Specifically, the coordinates of the sampling point with the greatest similarity to the building feature line of the image may be used as the location information of the mobile phone, and the posture information is determined by the building feature line of the image and the angle with the highest similarity in the sampling point code.

[0019] By jointly calculating the similarity of position and posture, and taking the position and posture of the sampling point with the highest similarity as the position and posture of the terminal, accurate 6DOF positioning can be achieved on the terminal, improving the user experience.

[0020] Based on the first aspect, in a possible embodiment, the acquiring of building outline features in a picture taken by a terminal includes: performing feature extraction on the picture taken by the terminal to acquire building outline features in the picture.

[0021] Based on the first aspect, in a possible embodiment, the feature extraction of the picture taken by the terminal to obtain the building outline features in the picture includes: performing morphological processing on the picture to eliminate the occlusion of the building outline features by non-target objects to obtain a processed picture; performing building semantic extraction on the processed picture to obtain the building outline features.

[0022] It can be seen that in the embodiment of the present application, before extracting the building outline features, considering the common existence of non-target objects such as green belts and road signs around urban buildings, which may cause occlusion of the building target of interest, resulting in the inability to accurately extract the building outline. Therefore, the embodiment of the present application adopts the closed operation in the field of computer vision to eliminate the occlusion problem by filling the gaps caused by the occluded area. In terms of building contour line extraction, the embodiment of the present application uses deep learning to perform building semantic extraction, realizes the accurate extraction of building contour lines, eliminates non-effective information, and assists in the extraction of Lupin's building contour features. Thereby, a solid foundation is provided for the present application to obtain 6DOF information with high precision and completeness.

[0023] Based on the first aspect, in a possible embodiment, obtaining multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature includes: receiving the multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature sent by a server.

[0024] That is to say, when the method described in the present application is jointly implemented through the interaction of the terminal and the server, the terminal can realize the joint positioning of position and posture through interactive communication with the server, the server executes the construction and storage of the building outline feature library, and downloads all the information of the building outline feature library or part of the information in the building outline feature library to the terminal, and the terminal then executes the extraction of outdoor building outline features based on the picture, matches the outdoor building outline features, to obtain the 6DOF information of the terminal.

[0025] Based on the first aspect, in a possible embodiment, obtaining multiple simulated building outline features and simulated pose information corresponding to each simulated building outline feature includes:

[0026] A three-dimensional model (3D model) is obtained based on a plurality of satellite images, and the three-dimensional model includes a plurality of simulated buildings; a road layer is determined in the three-dimensional model; a plurality of sampling points are determined in the road layer; and at each of the plurality of sampling points, the plurality of simulated buildings are respectively projected and encoded according to a plurality of simulated posture information to obtain contour features of the plurality of simulated buildings; each contour feature of the simulated building corresponds to a type of simulated posture information.

[0027] It can be seen that the embodiment of the present application can construct a building outline feature library on a large scale at low cost by extracting and encoding the simulated building outlines of the 3D model based on satellite images. And the data volume of the building outline feature library constructed by encoding is small. By implementing the present application, 6DOF positioning services can be provided as long as there are buildings in the area. Therefore, compared with traditional posture acquisition methods (such as posture detection hardware equipment that comes with mobile phones, large-scale professional posture measurement equipment to build point cloud feature libraries, etc.), it can have both high positioning accuracy and low cost advantages, saving manpower, material resources, financial resources and time costs.

[0028] In a second aspect, an embodiment of the present application provides a device for determining a terminal posture, characterized in that it includes:

[0029] The first acquisition module is used to acquire multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature, wherein the simulated building outline feature is obtained based on satellite images. The second acquisition module is used to acquire building outline features in a picture taken by a terminal. The posture determination module is used to determine the simulated posture information corresponding to the simulated building outline feature that matches the building outline feature among the multiple simulated building outline features as the posture information of the terminal.

[0030] That is to say, the device can be used to implement the method described in the first aspect.

[0031] Based on the second aspect, in a possible embodiment, the posture determination module is used to: match the building outline feature with the multiple simulated building outline features to obtain a simulated building outline feature that matches the building outline feature among the multiple simulated building outline features.

[0032] Based on the second aspect, in a possible embodiment, the posture determination module is also used to: obtain the initial position information and initial posture information of the terminal, wherein the initial position information is obtained by collecting sensors in the terminal; select a part of simulated building outline features from multiple simulated building outline features based on the initial position information and initial posture information of the terminal; match the building outline features with the part of simulated building outline features; and use the simulated building outline feature with the highest degree of matching among the part of simulated building outline features as the simulated building outline feature that matches the building outline feature.

[0033] Based on the second aspect, in a possible embodiment, the posture determination module is used to: calculate the initial position information and initial posture information of the building outline feature, and the similarity between the model position information and simulated posture information corresponding to each simulated building outline feature in the part of the simulated building outline features; the simulated building outline feature with the greatest similarity is used as the simulated building outline feature with the highest degree of matching.

[0034] Based on the second aspect, in a possible embodiment, the second acquisition module is used to:

[0035] Feature extraction is performed on the picture taken by the terminal to obtain the outline features of the building in the picture.

[0036] Based on the second aspect, in a possible embodiment, the second acquisition module is used to: perform morphological processing on the image to eliminate the occlusion of the building contour features by non-target objects to obtain a processed image; perform building semantic extraction on the processed image to obtain the building contour features.

[0037] Based on the second aspect, in a possible embodiment, the first acquisition module is used to: receive the multiple simulated building outline features and the simulated posture information corresponding to each simulated building outline feature sent by the server.

[0038] Based on the second aspect, in a possible embodiment, the first acquisition module is used to: acquire a three-dimensional model, the three-dimensional model is obtained based on multiple satellite images, and the three-dimensional model includes multiple simulated buildings; determine a road layer in the three-dimensional model; determine multiple sampling points in the road layer; at each of the multiple sampling points, respectively, perform projection encoding on the multiple simulated buildings according to multiple simulated posture information to obtain the multiple simulated building contour features; each simulated building contour feature corresponds to a type of simulated posture information.

[0039] In a third aspect, an embodiment of the present invention provides another device, the device comprising: a memory, a communication module and a processor; the memory is used to store instructions, the processor is used to execute the instructions, in some implementations, the device is a terminal, and the communication module is used to communicate with other devices (such as a server) or with a positioning satellite under the control of the processor. When the processor executes the instructions, the method steps described in any embodiment of the first aspect are performed.

[0040] In a fourth aspect, an embodiment of the present invention provides another device, the device comprising: a memory, a communication interface and a processor; the memory is used to store instructions, the processor is used to execute the instructions, in some implementations, the device is a server, and the communication interface is used to communicate with other devices (such as terminals) under the control of the processor. When the processor executes the instructions, the method steps described in any embodiment of the first aspect are performed.

[0041] In a fifth aspect, an embodiment of the present invention provides a readable non-volatile storage medium storing computer instructions, wherein the readable non-volatile storage medium includes computer instructions, wherein: the computer instructions are executed to implement the method described in the first aspect.

[0042] In a sixth aspect, an embodiment of the present invention provides a computer program product, which, when running on a computer, is executed to implement the method described in the first aspect.

[0043] It can be seen that the embodiments of the present application mainly achieve low-cost and high-precision acquisition of the terminal's 6DOF information from two aspects. On the one hand, the embodiments of the present application provide a low-cost data source, and the data source of the present application is a 3D model constructed from satellite images. Based on the 3D model, effective information is extracted to construct a building outline feature library for 6DOF positioning calls. On the other hand, the present application can provide complete and accurate position information and attitude information. In order to solve the problem that the overhead perspective of satellite images and the head-on perspective of terminal images are difficult to correctly match, the present application extracts the building outlines in the picture to match them with the building outline feature library, thereby obtaining high-precision position and angle information, thereby providing complete position information and attitude information (i.e., 6DOF information) for subsequent AR / VR application services, and improving the user experience of application services. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of the architecture of a terminal, a schematic diagram of the architecture of a server, and a schematic diagram of a system formed by the two in a terminal-server interaction scenario provided in an embodiment of the present application.

[0046] Figure 3 It is a flowchart of a method for determining a terminal posture provided by an embodiment of the present application;

[0047] Figure 4 It is a flowchart of another terminal posture determination method provided in an embodiment of the present application;

[0048] Figure 5 This is a schematic diagram of a scenario implementation process provided by an embodiment of the present application;

[0049] Figure 6 This is a schematic diagram of a scenario implementation provided by an embodiment of the present application;

[0050] Figure 7 It is a flowchart of another terminal posture determination method provided in an embodiment of the present application;

[0051] Figure 8 This is a schematic diagram of a scenario implementation regarding the attitude angle of a mobile phone provided in an embodiment of the present application;

[0052] Fig. 9 This is a schematic diagram of a scenario implementation provided by an embodiment of the present application;

[0053] Fig.10 It is a schematic diagram of some practical application scenarios provided by the embodiments of the present application;

[0054] Fig.11 It is a schematic diagram of some practical application scenarios provided by the embodiments of the present application;

[0055] Fig.12 It is a flowchart of another terminal posture determination method provided in an embodiment of the present application;

[0056] Fig.13 This is a schematic diagram of a scenario implementation provided by an embodiment of the present application;

[0057] Fig.14 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0058] Fig.15 It is a structural diagram of a server provided in an embodiment of the present application;

[0059] Fig.16 It is a structural schematic diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0061] In this article, the so-called "terminal posture" or "terminal posture information" refers to the terminal's position information and posture information, that is, the terminal's 6DOF information. In order to achieve low-cost and high-precision acquisition of the terminal's 6DOF information, each embodiment of the present application provides a terminal posture determination method. In the method described in the present application, on the one hand, a building outline feature library can be constructed, and on the other hand, the actual outdoor building outline features in the image can be extracted. On the other hand, the terminal's position and posture are jointly located based on the building outline feature library and the outdoor building outline features to obtain the terminal's 6DOF information.

[0062] The execution subject of the terminal posture determination method described in the embodiment of the present application can be a terminal or a server. In addition, the method can also be implemented through the interaction between the terminal and the server.

[0063] The terminal may be a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), a wearable device (such as a smart bracelet, a smart watch, etc.), a dedicated AR device, a dedicated VR device, a camera device (such as a video recorder, a smart camera, a digital camera, a video camera, etc.) or other devices that need to obtain 6DOF information. In some examples, the terminal may have a camera, a satellite positioning module, and an inertial sensor device.

[0064] The server can be an independent server, or a server in a cloud platform, a data center, or a server cluster. The server type can be, for example, a database server, an application server, a general server, a dedicated server, and the like.

[0065] When the method described in the present application is executed by a terminal alone, in one implementation, the terminal may execute a method for processes such as constructing a building outline feature library, extracting outdoor building outline features based on images, matching outdoor building outline features with the building outline feature library to obtain 6DOF information of the terminal. In another implementation, the terminal may obtain a building outline feature library in advance and save it locally, and the terminal may execute a method for processes such as extracting outdoor building outline features based on images, matching outdoor building outline features with the building outline feature library to obtain 6DOF information of the terminal (which may be referred to as a client offline mode).

[0066] When the method described in the present application is executed by a server alone, the server can execute methods such as constructing a building outline feature library, extracting outdoor building outline features based on pictures, matching outdoor building outline features with the building outline feature library to obtain 6DOF information of the terminal (which may be referred to as server offline mode).

[0067] When the method described in this application is implemented through interaction between a terminal and a server, such as Figure 1 The system architecture in which the terminal interacts with the server is shown. The terminal can realize the joint positioning of position and posture and obtain 6DOF information through interactive communication with the server (also known as client-server mode).

[0068] In one implementation, the server constructs and stores a building outline feature library, and downloads all the data in the building outline feature library or part of the data in the building outline feature library (such as the data in the retrieval buffer described later) to the terminal, and the terminal executes a method for extracting outdoor building outline features based on images, matching outdoor building outline features with the building outline feature library to obtain the terminal's 6DOF information, and other processes.

[0069] In another implementation, the server may pre-build a building outline feature library. After the terminal sends the picture to the server, the server extracts the outdoor building outline features based on the picture and matches the outdoor building outline features with the building outline feature library to obtain the terminal's 6DOF information, and sends the 6DOF information to the required terminal.

[0070] See also Figure 2 , Figure 2 An architectural diagram of a terminal, an architectural diagram of a server, and a system formed by the two in a scenario of interaction between the terminal and the server are exemplarily given.

[0071] like Figure 2 As shown, the terminal may include a hardware abstraction layer, a hardware abstraction layer data interface, a framework layer, and an upper application layer.

[0072] The embodiment of the present application can deploy the building outline extraction module 3 and the 6DOF determination module 4 in the framework layer of the terminal's operating system, and optionally, also includes an image correction module 2 and an initial posture module 1. These modules exist, for example, in the form of a binary software package for positioning applications, as system services, and are used by application layer services through an application programming interface (API). In addition, a database 5 is also configured in the terminal for storing the simulated building outline feature information from the server. The relevant description of the terminal is as follows:

[0073] Hardware abstraction layer: For example, it includes built-in satellite positioning modules (such as GPS positioning, GLONASS positioning, Galileo positioning, Beidou positioning, etc.), magnetometers, gyroscopes, cameras, chips and other hardware. The drivers and data reading and writing corresponding to these hardware are implemented through the hardware abstraction layer data interface, and the data and control interaction is carried out with the upper-level positioning application service program (such as AR / VR applications, etc.) according to the standard system interface.

[0074] Hardware abstraction layer data interface: Information can be obtained from the standard API interface of the hardware abstraction layer of the terminal operating system, such as sensor measurement information (such as magnetometer parameters, gyroscope parameters, etc.), satellite positioning information, and image data collected by the camera.

[0075] Initial posture module 1: used to obtain the initial position information and initial posture information of the terminal. The initial position information is provided by the satellite positioning module in the hardware abstraction layer, and the initial posture information is provided by the magnetometer and gyroscope in the hardware abstraction layer. In the specific implementation, the absolute direction estimated by the magnetometer and the relative direction estimated by the gyroscope can be combined through the filtering module to eliminate noise and obtain relatively stable posture information.

[0076] Image correction module 2: Since the image data obtained from the hardware abstraction layer can be taken from any angle and may not be in the same perspective as the simulated building contour features in the database 5, the image correction module 2 is used to construct a rotation matrix based on the posture information obtained from the initial posture module 1, and then restore the image data to the observation perspective corresponding to the simulated building contour features, so as to perform the similarity matching operation in the subsequent 6DOF determination module 4.

[0077] Building outline extraction module 3: This module can be used to extract building outline features (also called building outline features, building outlines, or building outline information) from pictures captured by the camera or pictures sent to this terminal by other devices. The main processing processes include, for example, building semantic extraction based on deep learning, geometric straight line extraction, and building outline extraction and optimization that integrates semantic and geometric information. This module can avoid interference from occlusion and severe weather conditions and output complete building outline features.

[0078] 6DOF determination module 4: This module can be used to retrieve the closest sampling point of the building outline in the image among multiple simulated building outline features in the database 5 to obtain 6DOF information. Its main processing includes initial position and posture estimation, similarity matching, taking the sampling point with the highest similarity as the final matching point, and outputting 6DOF information.

[0079] Database 5: used to store multiple building outline feature codes, each building outline feature code includes a unique code name, a simulated building outline feature (or building outline feature, or building outline, or building outline information), a corresponding latitude and longitude address (location information), and corresponding posture information. The multiple simulated building outline features are generated by the server and sent to the terminal for storage. The multiple simulated building outline features can be all the simulated building outline features corresponding to the 3D model constructed by the server based on the satellite image, for example, simulated building outline features of the order of magnitude of the city range or the order of magnitude of the regional range; or they can be part of the simulated building outline features corresponding to the 3D model, for example, simulated building outline features of the order of magnitude of the preset range based on the initial location information and initial posture information of the terminal. In some examples, the data stored in the database 5 is consistent with the data stored in the building outline feature library 7 in the server, that is, the server can send all the building outline feature codes in the building outline feature library 7 to the terminal and save them in the database 5 of the terminal. In some other examples, the data stored in the database 5 is part of the data stored in the building outline feature library 7 in the server, that is, the server can send a part of the building outline feature codes in the building outline feature library 7 to the terminal and save it in the database 5 of the terminal.

[0080] Application layer: used to provide various applications, such as various AR / VR application services, including dedicated positioning software, various e-commerce shopping applications, various social communication application software, various car application software, O2O door-to-door service application software, exhibition hall self-guided tour application, family anti-lost application software, emergency rescue service software, audio and video entertainment software, game software, etc., which provide accurate positioning location information. Typical application scenarios include AR navigation.

[0081] like Figure 2 As shown, the server may include a simulated building outline feature construction module 6 and a building outline feature library 7, which are described as follows:

[0082] Simulated building outline feature construction module 6: This module can be used to extract effective information from a large 3D model, such as concise and efficient feature encoding of the 3D model. Optionally, when there is no directly available 3D model, it can also be used to construct a 3D model based on satellite images (or satellite maps). The main processing process of the simulated building outline feature construction module 6 includes: sampling point sampling, building cylindrical projection, acquisition of building outline coding, etc. The content of the building outline feature coding includes the unique name of the code, simulated building outline features, corresponding latitude and longitude addresses (location information), corresponding posture information, etc.

[0083] Building outline feature library 7: used to store building outline feature codes from simulated building outline feature construction module 6, such as simulated building outline features at the city level or regional level. This application does not limit the geographical level.

[0084] It should be noted that Figure 2 It is only used to exemplify possible implementation schemes of this application. In practical applications, Figure 2 It may also include more or fewer functional modules, more or fewer hardware components. Alternatively, the technical ideas / technical solutions based on the embodiments of the present application may also be implemented in other forms of terminal architectures and server architectures, which are not specifically limited in the present application.

[0085] For the sake of convenience, the various method embodiments described below are expressed as a combination of a series of action steps, but those skilled in the art should be aware that the specific implementation of the technical solution of the present application is not limited by the order of the described series of action steps.

[0086] See also Figure 3 , Figure 3 The present invention provides a flow chart of a method for determining a terminal posture according to an embodiment of the present invention. The method can be applied to a terminal or a server, and the method includes but is not limited to the following steps:

[0087] S11, obtaining multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature.

[0088] In one implementation, when the execution subject of the present method is a terminal, the terminal can obtain the relevant data of the building outline feature library in advance, and the relevant data in the building outline feature library may include multiple simulated building outline features and the simulated posture information corresponding to each simulated building outline feature. For example, the terminal can receive the building outline feature library sent by the server, and save the relevant data in the building outline feature library locally (for example, in the database of the terminal). In addition, the terminal can also obtain the relevant data in the building outline feature library by other means (for example, obtained through a USB flash drive, or sent by other devices). In this way, when the terminal needs to execute the terminal posture determination method provided in the embodiment of the present application, the multiple simulated building outline features and the simulated posture information corresponding to each simulated building outline feature are obtained locally.

[0089] In another implementation, when the execution subject of the method is a terminal, the terminal can obtain the relevant data of the building outline feature library in real time in an online manner, and the relevant data in the building outline feature library may include multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature. For example, when the terminal needs to execute the terminal posture determination method provided in the embodiment of the present application, the server is requested to send the building outline feature library, so as to obtain the multiple simulated building outline features and the simulated posture information corresponding to each simulated building outline feature.

[0090] In another implementation, when the execution subject of the method is a server, the server can pre-build a building outline feature library, and the relevant data in the building outline feature library may include multiple simulated building outline features and simulated pose information corresponding to each simulated building outline feature. When the server receives the picture (or the picture and the initial pose information of the terminal) and the 6DOF service request sent by the terminal, the server obtains the multiple simulated building outline features and the simulated pose information corresponding to each simulated building outline feature from the local.

[0091] Among them, the simulated building outline features are obtained based on the 3D model of the satellite image. The 3D model based on the satellite image is a highly abstract simple model, which uses wireframes to represent the building information in the city. There are a large number of line features in the 3D model, such as building outlines, urban landscape outlines, road outlines, etc., which can provide a basis for the matching of terminal images. Generally speaking, urban landscape outlines are changeable and road outlines are severely blocked, and only building outlines are the most stable. Therefore, the embodiment of the present application uses a 3D model to extract building outline features (building outline features are, for example, building outlines), thereby constructing a building outline feature library.

[0092] In this paper, the building outline features extracted from the 3D model can also be called simulated building outline features.

[0093] In addition, unless otherwise specified in this article, the so-called "building outline features" may specifically refer to "building outlines".

[0094] S12: Obtain building outline features in the image taken by the terminal.

[0095] In one implementation, when the execution subject of the method is a terminal, the terminal that executes the method of the application and the terminal that takes the picture can be the same terminal.

[0096] In another implementation, when the execution subject of the method is a terminal, the terminal executing the method of the application and the terminal taking the picture may not be the same terminal. For example, one terminal may assist another terminal in implementing the 6DOF positioning described in the embodiment of the application.

[0097] In yet another implementation, when the execution subject of the method is a terminal, the terminal may perform feature extraction on the image, thereby obtaining building outline features of the buildings in the image.

[0098] In another implementation, when the execution subject of the method is a server, the server may perform feature extraction on the image provided by the terminal, thereby obtaining building outline features of the buildings in the image.

[0099] S13: Match the building outline feature with a plurality of simulated building outline features to obtain a simulated building outline feature that matches the building outline feature among the plurality of simulated building outline features.

[0100] In the embodiment of the present application, the building outline features in the image and the multiple simulated building outline features in the building outline feature library are both two-dimensional coded data, and the two are matchable.

[0101] Specifically, when the execution subject of the method is a terminal or a server, the terminal or the server performs similarity matching processing on the building outline feature and the multiple simulated building outline features, thereby obtaining a simulated building outline feature that matches the building outline feature among the multiple simulated building outline features. For example, among the multiple simulated building outline features, the one with the highest similarity to the building outline feature of the image is taken as the "simulated building outline feature that matches the building outline feature".

[0102] S14. Determine the simulated posture information corresponding to the matched simulated building outline feature as the posture information of the terminal that photographed the image.

[0103] The simulated position information corresponding to the simulated building outline features includes simulated position information and simulated attitude information, the simulated position information represents the specific latitude and longitude information of a specific position point (such as the sampling point described later) in the 3D model based on the satellite map, and the simulated attitude information includes the attitude information when the building outline features are collected at a specific position point (such as the sampling point described later) in the 3D model based on the satellite map. The simulated position information and simulated attitude information have high accuracy. The simulated position information and simulated attitude information are bound to the simulated building outline features and stored in the building outline feature library.

[0104] The simulated position information corresponding to the matched simulated building outline features is determined as the position information of the terminal that takes the image, that is, the simulated position information corresponding to the matched simulated building outline features is used as the position information of the terminal that takes the image, and the simulated posture information corresponding to the matched simulated building outline features is used as the posture information of the terminal that takes the image, so that the terminal can obtain high-precision and complete 6DOF information in real time.

[0105] It can be seen that the embodiments of the present application mainly achieve low-cost and high-precision acquisition of the terminal's 6DOF information from two aspects. On the one hand, the embodiments of the present application provide a low-cost data source, and the data source of the present application is a 3D model constructed from satellite images. Based on the 3D model, effective information is extracted to construct a building outline feature library for 6DOF positioning calls. On the other hand, the present application can provide complete and accurate position information and attitude information. In order to solve the problem that the overhead perspective of satellite images and the head-on perspective of terminal images are difficult to correctly match, the present application extracts the building outlines in the picture to match them with the building outline feature library, thereby obtaining high-precision position and angle information, thereby providing complete position information and attitude information (i.e., 6DOF information) for subsequent AR / VR application services, and improving the user experience of application services.

[0106] The implementation process of the above S11 is further described below. Figure 4 and Figure 5 In one embodiment, the process of obtaining multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature may include but is not limited to the following steps:

[0107] S111. Obtain a 3D model (3D Model, i.e., three-dimensional model) according to a satellite image.

[0108] For example, a WorldView-03 satellite image of a certain area can be obtained, and the ground resolution under the satellite is 0.35m. By using multiple (for example, three) satellite images with different viewing angles, a stereo pair can be formed, and then a 3D model can be constructed based on the stereo pair. Figure 5 (1) is a schematic diagram of a satellite image. Figure 5 (2) is a schematic diagram of a portion of the content of a 3D model image generated based on multiple satellite images.

[0109] Since satellite images cover a large area, they can be used to quickly generate large-scale city-level or regional-level 3D model data. Compared with traditional 3D point cloud feature libraries, 3D models built based on satellite images can greatly reduce model construction costs.

[0110] It should be noted that the geographical range level of the 3D model can also be hundreds of meters, several kilometers, etc., which is not limited here.

[0111] In addition, in some implementations, this step S111 is optional, that is, the input data of the embodiment of the present application can be directly a 3D model constructed based on a satellite map.

[0112] S112. Determine a road layer in the 3D model.

[0113] In one example, the bottom area of ​​the 3D model can be first identified as the ground, and the area outside the 3D building area can be identified as the road. For example, the road layer can be identified using the 3D model ground plane estimation method. Then, the non-road areas such as flower beds, lawns, and ponds can be identified using the deep learning method and removed from the road layer, thereby obtaining a complete road layer.

[0114] S113. Determine a plurality of candidate sampling points in the road layer.

[0115] In order to fully simulate different locations in the city, multiple locations in the road layer can be defined as candidate sampling points, and the unique number and longitude and latitude coordinates of each sampling point can be recorded. For example, grid sampling (Grid modeling) can be performed on the road layer at 3-meter intervals to obtain a sampling point layer, such as Figure 5 As shown in (3) in .

[0116] S114, performing cylindrical projection encoding on each candidate sampling point, so as to obtain a plurality of simulated building outline features and simulated pose information corresponding to each simulated building outline feature.

[0117] Since there are a large number of line features in the 3D model based on satellite images, such as building outlines, urban landscape outlines, road outlines, etc., it can provide a feature library basis for the matching of terminal images. However, urban landscapes are changeable and roads are severely blocked, so only building outlines are the most stable.

[0118] Therefore, the embodiment of the present application can extract the building contour line in the 3D model at different sampling points with different postures (angles). The building contour line in the 3D model can also be called the simulated building contour feature. The latitude and longitude coordinates of the sampling point are the position information corresponding to the simulated building contour feature, and the sampling posture (sampling angle) of the sampling point is the posture information corresponding to the simulated building contour feature. Then, a building contour feature library can be constructed based on a large number of simulated building contour features and the corresponding position information and posture information.

[0119] In a specific implementation, for each candidate sampling point of the road layer, a cylindrical projection surface is established with the sampling point as the center, and then a 360-degree building outline code of the 3D model is generated.

[0120] For example, a 360-degree virtual cylindrical projection surface with a radius of 1 m is established with each sampling point as the center, and the surrounding three-dimensional buildings are projected onto the cylindrical projection surface in a perspective projection manner. Figure 5 (4) in describes the process of building encoding centered on a sampling point. Figure 5 The cylinder in (4) represents the projection surface. The outlines of the surrounding buildings can be projected onto this cylindrical surface in the manner of projection imaging to form a curve.

[0121] See also Figure 5 In (5), after the cylindrical projection surface is unfolded, a two-dimensional line feature coding map with a horizontal coordinate of 0-360 degrees and a vertical coordinate of the building outline is formed. Among them, 0 degrees represents the north direction, and clockwise represents the angle with the north direction. Since the cylindrical projection has the characteristics of being larger near and smaller far away, the coding features of the same building at different sampling points will show different slopes, different coding angles, and different coding heights. The slope, coding angle, and coding height corresponding to each sampling point represent the posture information of each sampling point.

[0122] In this article, the position information of the sampling point can also be called simulated position information, the posture information of the sampling point can also be called simulated posture information, or in other words, the posture information of the sampling point can also be called simulated posture information.

[0123] In addition, in order to restore the occlusion relationship, when performing projection, it can be assumed that a ray is emitted from the sampling point, and only the first intersection of the ray and the building is recorded, and the subsequent intersections are considered as occlusions and removed. In this way, the building contours of all sampling points and the corresponding position information and posture information can be saved together in the building contour feature library.

[0124] Below Figure 6 Take the cylindrical projection coding as an example to further explain, Figure 6 The coding diagram of the building outline at different sampling points is shown. The coding on the right is the building outline coding obtained by cylindrical projection from three points A, B, and C. The coding is 0 degrees in the north direction, and the clockwise direction is positive, indicating the angle at which the building appears and the height of the projection. Point A is closer to building P1 and farther from building P2 than point B, so the coding of P1 is larger and higher at point A, and the coding of P2 is smaller and shorter at point A. At the same time, the angles at which the buildings appear are also different. The coding of building P1 at point A appears earlier than that at point B, and building P2 appears later than that at point B. Similarly, point C is farther away from building P1 than point A, and closer to building P2. Therefore, the coding of building P1 at point C is smaller and shorter than that at point A, and the coding of building P2 is larger and higher. In terms of angle, C and A have similar initial perspectives for building P1, so they appear almost at the same time, but the perspectives for seeing the right side of P1 are different, resulting in differences in coding angles.

[0125] It should be noted that, in a possible implementation, the embodiment of the present application may also use other methods (rather than cylindrical projection) for projection coding at the sampling point, such as fisheye projection. Thus, multiple simulated building outline features and simulated pose information corresponding to each simulated building outline feature can also be obtained.

[0126] It can be seen that the embodiment of the present application provides a 6DOF positioning method for a 3D model library based on satellite images. By extracting and encoding the simulated building outlines of the 3D model, a building outline feature library can be constructed at low cost and on a large scale. And the data volume of the building outline feature library constructed by encoding is small. By implementing this application, 6DOF positioning services can be provided as long as there are buildings in the area. Therefore, compared with traditional posture acquisition methods (such as posture detection hardware equipment that comes with mobile phones, large-scale professional posture measurement equipment to build point cloud feature libraries, etc.), it can have both high positioning accuracy and low cost advantages, saving manpower, material resources, financial resources and time costs. For example, compared with the existing point cloud feature library acquisition solution, the cost is shortened from tens of millions to hundreds of thousands, the modeling time is shortened from several months to a few weeks, and the amount of data is reduced from PB level to GB level, greatly reducing manpower, material resources and time costs.

[0127] In addition, with the acceleration of urbanization, the development of 5G networks and high-performance mobile devices, the demand for urban navigation and urban VR / AR has increased rapidly. Buildings are the most prominent features of cities. However, urban buildings belong to built-up areas and usually do not change significantly. Choosing buildings as references for obtaining terminal position and posture has significant practical significance. Therefore, the solution of constructing a building contour feature library has more practical value and broad application prospects.

[0128] The implementation process of the aforementioned S12 is further described below. Figure 7 In one embodiment, the process of obtaining the outline features of a building in a picture taken by a terminal may include but is not limited to the following steps:

[0129] S121. Perform correction operations on the pictures taken by the terminal.

[0130] In order to make the pictures taken by the terminal at any angle usable for positioning, the pictures need to be preprocessed, that is, the picture correction operation. The purpose of picture correction is to restore the pictures taken at any angle to the situation where the yaw angle is 0 degrees and the roll angle is 90 degrees, so as to transform them into the same posture and perspective as the encoding in the building outline of the 3D model (i.e., simulate the building outline features), so as to facilitate the subsequent smooth implementation of outline extraction and feature matching.

[0131] Taking a mobile phone as an example, the attitude angle of the mobile phone is defined as Figure 8As shown, it is composed of three angles: pitch angle (pitch), yaw angle (yaw), and roll angle (roll). Figure 8 In the figure, the front view and the bird's-eye view of the mobile phone are examples of observing the mobile phone from different angles, respectively. The short axis of the mobile phone points to the right to represent the x-axis, the long axis points upward to represent the y-axis, and the z-axis points perpendicular to the screen to the user. The pitch angle represents the rotation around the x-axis, the heading mark represents the rotation around the z-axis, and the roll angle represents the rotation around the y-axis. The angle definition of this part is a well-known method. It will not be expanded in detail here.

[0132] For example, when you need to convert a picture taken from any perspective to a yaw angle of 0 degrees and a roll angle of 90 degrees, you only need to follow the rules of perspective projection and calculate the image pixel coordinates from the corresponding projection matrix to the imaging plane. The projection formula from the image plane coordinate system to the world coordinate system is shown in formula (1):

[0133]

[0134] Where X, Y, Z represent the coordinates in the world coordinate system, x, y represent the imaging coordinates of X, Y, Z in the image plane coordinate system, f represents the focal length, c x , c y represents the image principal point, and t is the translation matrix.

[0135] Where R represents the rotation matrix, which can be constructed from the above three attitude angles, as shown in formula (2):

[0136]

[0137] During image correction, the world coordinates, camera internal parameters, and translation matrix remain unchanged. Only the attitude angle changes. Therefore, after the attitude angle of the mobile phone is calculated using the gyroscope and magnetometer, it is converted to a yaw angle of 0 degrees and a roll angle of 90 degrees. The image plane coordinates can be obtained using formula (3):

[0138]

[0139] It should be noted that, in some implementations, S121 is an optional step.

[0140] S122, performing morphological processing on the image.

[0141] In some embodiments, the image processed in step S11 may be further subjected to morphological processing.

[0142] In an embodiment of the present application, the building outline feature library constructed by the 3D model is a complete building outline obtained based on the 3D building projection, and is not blocked by non-target objects such as urban greening, urban landscape, etc.

[0143] In the real environment, buildings are usually blocked by non-target objects such as urban greenery and urban landscape around them, which poses a more severe challenge to extracting the complete building outline straight line. In order to achieve the matching of building outlines, in the embodiment of the present application, it is necessary to remove the occlusion of non-target objects in the picture taken by the terminal, so as to facilitate the subsequent extraction of the complete building outline and achieve accurate matching with the building outline feature library.

[0144] In real environments, urban greenery / urban landscapes such as trees, signboards, street lights, and electric poles extend beyond the skyline of buildings in the image, blocking the edges of building outlines and interfering with the shape of building outlines. The embodiment of the present application uses morphological operation closing operations to eliminate the occlusion problem to avoid the influence of trees and the like on subsequent straight line extraction.

[0145] Specifically, the morphological operation closing operation brightens the dark grayscale areas of the image pixels by first dilating and then corroding. Fig. 9 As shown, Fig. 9 The process of morphological closing operation and the expected effect are described. The closing operation mainly includes two parts: dilation operation and erosion operation. Fig. 9 The above pixel grid diagram represents the target image, the white area represents the background area, the gray area represents the building, and the broken area represents the occluded object. The dilation and erosion operations construct the structural element operator and perform convolution operations with the target image in turn. The dilation operation replaces the value of the target pixel with the local maximum value of the convolution coverage area, expands the bright area, and achieves multiple connected areas. On the contrary, erosion calculates the local minimum value within the coverage of the convolution kernel, reduces the bright area, and eliminates small spots. In this way, objects such as tree branches that appear outside the skyline (i.e. in the sky) become bright, achieving the purpose of occlusion culling.

[0146] The following uses an actual application scenario as an example to intuitively present the processing effect of the closing operation. Fig.10 As shown, Fig.10 The comparison in the example shows that the occluded branches are brightened to achieve the effect of removing the occlusion. Fig.10 (a) and (b) are two original pictures. Fig.10 (c) and (d) in the figure are the results of closing the two original images. By comparing them, we can see that Fig.10 The branches in (a) are Fig.10 In (c), the image has been removed, revealing the complete building outline. Fig.10 The branches in (b) are Fig.10 The image in (d) is also eliminated, revealing the outline of the building.

[0147] S123, extracting building semantics from the image to obtain building outlines.

[0148] In some application scenarios, even if the tree occlusion is removed, the influence of the clutter lines inside the building and on the ground cannot be eliminated. Therefore, the embodiment of the present application can completely identify the building by extracting the semantic information of the building, while keeping the edges of the building smooth and complete, thereby assisting in the subsequent extraction of the building contour lines.

[0149] Building semantic information extraction is essentially a segmentation problem, that is, the boundary line between the segmented building and the sky. The embodiment of the present application can realize the extraction of building semantic information by the method of deep learning. The deep learning method can be used for the classification and segmentation tasks of natural images. Its multi-layer nested chain structure can theoretically simulate any function and has a strong fitting ability. In the deep network for segmentation tasks, for example, the deeplab-v3 network can be selected, in which the ResNet network structure can avoid gradient vanishing or gradient explosion while realizing ultra-deep network training by skipping connection, and its advanced atrous convolution operation can increase the receptive field, avoid excessive reduction of image resolution and loss of spatial context information. In addition, the hollow space pyramid pooling (ASPP) structure adopted by it can effectively extract multi-scale line features, providing a more solid theoretical basis for building contour extraction.

[0150] The following is an example of a practical application scenario to intuitively present the processing effect of building semantic extraction. Fig.11 As shown, Fig.11 (a) and (b) in the figure describe the results of the traditional contour extraction algorithm on two images. Fig.11 (c) and (d) are the processing results of the two images by the deep learning semantic segmentation method in the embodiment of the present application. Fig.11 (e) and (f) are the results of building outline extraction in the embodiment of the present application.

[0151] Traditional contour extraction algorithms usually use the LSD line extraction algorithm. However, when applied to practice, this algorithm cannot distinguish the straight lines of buildings and other objects, resulting in the extraction of a large number of disordered line segment information.

[0152] The embodiment of the present application processes the two images by using a deep learning semantic segmentation method, and can better extract the complete buildings and building contours, such as Fig.11 The bold black lines in (e) and (f) represent the extracted building contours.

[0153] The following is a discussion of the extraction process of the building outline in conjunction with steps S122 and S123. The embodiment of the present application first performs a closing operation on the picture taken by the terminal to remove the occlusion of obstacles, and then uses deep learning semantic segmentation to identify the boundary between the sky and the building (i.e., the semantic segmentation line of the building) in the semantic segmentation result map, and uses the boundary as a constraint to eliminate other line segments that are not at the junction of the sky and the building. For example, the LSD straight line extraction algorithm can be used to quickly extract the line segment features in the picture. Then, the distance between each line segment and the semantic segmentation line is calculated, and those that exceed the threshold are regarded as interfering line segments and eliminated. Since LSD will extract repeated short line ends, distance clustering is performed and the longest line segment of the same type is extracted. Then, the retained straight line segments are extended along the gradient direction of the boundary of the building (i.e., the semantic segmentation line of the building), retaining the shape and gradient information until they are extended outside the building range. Finally, the extended line segments are connected end to end to form a complete building outline.

[0154] It can be seen that in the embodiment of the present application, before extracting the building outline features, considering the widespread existence of urban infrastructure such as green belts and road signs around urban buildings, the building targets of interest may be blocked, resulting in the inability to accurately extract the building outline. Therefore, the embodiment of the present application adopts closed operations in the field of computer vision to eliminate the occlusion problem by filling the gaps caused by the blocked area. In terms of building contour line extraction, the embodiment of the present application utilizes the powerful target recognition ability of deep learning to achieve accurate extraction of building contour lines, eliminate non-effective information, and assist in the extraction of Lupin's building contour features. Thereby, a solid foundation is provided for the present application to obtain 6DOF information with high precision and completeness.

[0155] The implementation process of the above S13 is further described below. Fig.12 In one embodiment, the process of matching the building outline feature with a plurality of simulated building outline features to obtain a simulated building outline feature that matches the building outline feature among the plurality of simulated building outline features may include but is not limited to the following steps:

[0156] S131. Obtain initial position information and posture information of the terminal.

[0157] In the present application embodiment, through the above Figure 4 The building outline feature library constructed in the embodiment may contain a large number of building outline codes. For example, the building outline feature library at the city level has a huge amount of data. In order to improve the subsequent search efficiency of these building outline codes and reduce the time consumption of matching obstacle outline features, the terminal sensor can be used to obtain initial position information and posture information, so as to establish a search buffer based on the initial position information and posture information.

[0158] S132. Determine a retrieval buffer according to the initial position information and the posture information, where the retrieval buffer includes a plurality of simulated building outline features.

[0159] In the embodiment of the present application, a search buffer is established based on the initial position information and posture information, and the search buffer includes a part of the building outline feature coding in the building outline feature library. That is, only a part of the coding data in the building outline feature library needs to be searched.

[0160] For example, the initial position of the terminal (i.e., precise and rough latitude and longitude information) can be determined based on the positioning module in the terminal, such as the Global Positioning System (GPS) positioning module, the GLONASS positioning module, the Galileo satellite navigation system (Galileosatellite navigation system) positioning module, the Beidou positioning module, etc. In addition, the initial attitude of the terminal (i.e., precise and rough attitude information) can be determined jointly based on the gyroscope and magnetometer in the terminal. Then, a circular buffer can be constructed within a preset range (e.g., a radius of 30 meters) with the initial position as the center of the circle, and the sampling points in the circular buffer are used as points to be matched. In the circular buffer, the initial retrieval angle (i.e., angle buffer) is constructed within the range of the attitude estimation value (e.g., plus or minus 15 degrees).

[0161] Specifically, from the previous Figure 7 It can be seen from the description of the embodiment that after the picture is corrected, the yaw angle of the picture has been transformed to a situation where the yaw angle is 0 degrees and the roll angle is 90 degrees. It is only necessary to calculate the accurate pitch angle to obtain the attitude information. The pitch angle information is obtained by calculating the angle between the building contour line of the picture and the corresponding segment in the code. The cameras of different terminals correspond to different field of view angles, for example, the field of view angle can be 70 degrees. Therefore, the building contour line in the acquired picture is encoded as 70 degrees. Subsequently, 70 degrees can be used as the matching length, and the Euclidean distance similarity of the building contour line of the picture and the feature library encoding can be matched in sequence with a step length of 0.2 degrees, and the angle with the highest similarity is recorded as the pitch angle. In order to avoid interference between building contour lines with similar angles at different angles in the feature encoding, the pitch angle estimated by the gyroscope and magnetometer is used as the initial pitch angle, and a pitch angle retrieval buffer of plus or minus 15 degrees is constructed. In this way, while eliminating interference, the retrieval efficiency is also improved.

[0162] S133: Perform similarity matching between the building outline features extracted from the image and the multiple simulated building outline features in the retrieval buffer to obtain the location information and posture information of the terminal.

[0163] Specifically, the points to be matched in the retrieval buffer and the various angles of the points to be matched can be traversed to determine the simulated building outline feature with the highest similarity to the building outline feature extracted from the image, and the position information (simulated position information) of the sampling point corresponding to the simulated building outline feature is used as the position information of the terminal, and the posture information (simulated posture information) of the sampling point corresponding to the simulated building outline feature is used as the posture information of the terminal.

[0164] In one example, the embodiment of the present application may use the Euclidean distance as a similarity criterion in the similarity matching to measure the similarity between the building feature line of the image and each building outline feature code in the retrieval buffer, and take the position and angle with the highest similarity as the posture information of the terminal's position information. Specifically, the coordinates of the sampling point with the greatest similarity to the building feature line of the image may be used as the position information of the mobile phone, and the posture information is determined by the building feature line of the image and the angle with the highest similarity in the sampling point code. Each sampling point in the building outline feature library represents a 360-degree building outline code (0 degrees is the north direction, and the clockwise direction indicates the angle between the direction and the north direction), while the building outline of the terminal's image only includes an angle interval, the size of which is determined by the terminal's field of view. Exemplarily, the Euclidean distance of each segment of the building outline of the image and the building outline feature code in the retrieval buffer is iteratively calculated with a step size of 0.5 degrees, and the angle with the highest similarity is taken as the direction of the image. The similarity measure is determined jointly by the position similarity and the angle similarity. Exemplarily, the similarity measure is calculated using the following formula (4):

[0165]

[0166] Among them, X and Y represent the building feature code of the terminal image and the building outline feature code in the retrieval buffer, respectively. i and i They represent the components in the building outline feature code respectively, and n represents the total number of components in a building outline feature code.

[0167] by Fig.14Taking the scenario shown as an example, a search buffer with a radius of 30 meters can be constructed according to the initial position of the terminal, and all sampling points in the search buffer are used as matching points to participate in the matching. In addition, an angle buffer is constructed according to the angle detected by the terminal's sensor, and the building feature code of the image is matched with the building contour feature code of the matching point in the search buffer with a step size of 0.2°. The position information of the matching point with the highest matching degree (for example, the highest similarity) is used as the position information of the terminal, and the pitch angle of the matching point is used as the pitch angle of the terminal. Finally, the pitch angle is combined with the yaw angle and the roll angle to form the complete attitude information of the terminal. In other words, through the above process, accurate 6DOF information of the terminal is obtained.

[0168] It can be seen that the embodiment of the present application makes full use of the hardware components in the existing terminal (such as positioning module, gyroscope and magnetometer) to obtain the initial position information and posture information, uses the camera to collect pictures and extract the building outline features, and constructs a retrieval buffer (initial position buffer) with the initial position information obtained by the positioning module, reducing the search range in the building outline feature library, and uses the initial posture information collected by the gyroscope and magnetometer to construct an initial angle buffer, assist in direction estimation, speed up retrieval efficiency, and achieve robust retrieval and similarity matching. By jointly calculating the position and posture similarity, the position and posture of the sampling point with the highest similarity are used as the position and posture of the terminal, and 6DOF positioning is achieved on the terminal. Practice has shown that the implementation of this embodiment can achieve a positioning accuracy of 3 meters and a posture accuracy of 3 degrees, which fully meets the requirements of AR / VR and other applications for posture accuracy.

[0169] The following is an overall description of the technical solution of this application using a practical application scenario.

[0170] The user can shoot the environment at any angle using the terminal at any location. After obtaining the picture, the application with 6DOF requirements initiates a 6DOF positioning service request, and the hardware components in the terminal (such as the positioning module, gyroscope and magnetometer) operate to obtain the initial position information and attitude information. The terminal extracts the building outline in the picture, searches and matches it with the building outline feature library, and retrieves the sampling point with the highest similarity in the library. The position information and attitude information corresponding to the sampling point are the current location and shooting angle of the user, thereby achieving real-time acquisition of 6DOF information on the terminal.

[0171] It can be seen that this application achieves high-precision and complete acquisition of 6DOF information while reducing data collection costs, fully meeting the requirements of AR / VR and other applications for posture accuracy and improving user experience.

[0172] The above describes in detail the method of the embodiment of the present application, and the following provides the related device of the embodiment of the present application.

[0173] See also Fig.14 , Fig.14 1 is a schematic diagram of a terminal 10 provided in an embodiment of the present application. The terminal 10 may include: a processor 101, a memory 102 (one or more computer-readable storage media), a communication module 103, and an input / output system 105. These components may communicate on one or more communication buses 104.

[0174] The input-output system 105 is mainly used to realize the interactive function between the terminal 10 and the user / external environment, and mainly includes the input-output device of the terminal 10. In a specific implementation, the input-output system 105 may include a touch screen controller 1051, an audio controller 1052, a sensor controller 1053, and a camera controller 1057. Among them, each controller can be coupled with its corresponding peripheral device (touch screen 1054, audio circuit 1055, sensor 1056, and camera 1058). Among them, the touch screen 1054 can be used to sense touch operations and display pictures, such as AR / VR pictures. The camera 1058 can obtain images by shooting. In a specific implementation, the sensor 1056 may include an accelerometer, a speedometer, a gyroscope, a magnetometer, etc., for monitoring the initial posture information of the terminal. It should be noted that the input-output system 105 may also include other I / O peripherals.

[0175] The processor 101 may be integrated to include: one or more CPUs, a clock module (not shown) and a power management module (not shown). The clock module is mainly used to generate clocks required for data transmission and timing control for the processor 101. The power management module is mainly used to provide stable and high-precision voltages for the processor 101, the communication module 103, and the input / output system 105.

[0176] The communication module 103 is used to receive and send information, and mainly integrates the receiver and transmitter of the terminal 10. In a specific implementation, the communication module 103 is optionally but not limited to: one or more of a radio frequency (RF) module, a Wi-Fi module (not shown), a Bluetooth module (not shown), etc. These modules can be used to communicate and interact with the server, such as receiving information from the server (such as multiple building outline feature codes), and for example, sending information to the server (such as pictures). The communication module 103 may also include a positioning module, such as a Global Positioning System (GPS) positioning module, a GLONASS positioning module, a Galileo satellite navigation system positioning module, a Beidou positioning module, etc. The positioning module is used to obtain the initial location information of the terminal. In some embodiments, the communication module 103 can be implemented on a separate chip.

[0177] The memory 102 is coupled to the processor 101 and is used to store various software programs and / or multiple sets of instructions. In a specific implementation, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 102 can store an operating system (hereinafter referred to as the system), such as an embedded operating system such as ANDROID, IOS, WINDOWS, or LINUX. The memory 102 can store application code and data, such as application code and data such as AR / VR. The memory 102 can also store a network communication program, which can be used to communicate with a server. The memory 102 can also store a user interface program, which can receive user control operations on the application through a graphical operating interface, such as an AR / VR interface, and through input controls such as menus, dialog boxes, and buttons.

[0178] Specifically, the memory 102 may also store codes of methods including the embodiments discussed herein, and / or codes of functional modules. The processor 101 is used to call the code in the memory 102 to execute the terminal posture determination method described in the embodiments of the present application.

[0179] It should be understood that the terminal 10 is only an example provided in the embodiment of the present application, and the terminal 10 may have more or fewer components than those shown, may combine two or more components, or may have different configurations of components.

[0180] See also Fig.15 , Fig.15 The schematic diagram of the structure of a server 60 provided in an embodiment of the present application. The server 60 in this embodiment includes: at least one processor 601, a communication interface 602 and a memory 603. The processor 601, the communication interface 602 and the memory 603 can be connected via a bus or other means, or partially or completely coupled together. The embodiment of the present application takes the connection via a bus 605 as an example. Among them,

[0181] The processor 601 may be a general-purpose processor, such as a central processing unit (CPU).

[0182] The communication interface 602 may be a wired interface (e.g., an Ethernet interface) or a wireless interface (e.g., a cellular network interface or a wireless local area network interface), and may be used to communicate with a terminal. In an embodiment of the present application, the communication interface 602 may be used, for example, to send multiple building outline feature codes to a terminal, or may be used to receive a picture sent by a terminal.

[0183] The memory 603 may include a volatile memory, such as a random access memory (RAM); the memory may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 603 may also include a combination of the above-mentioned types of memory. The memory 603 is used to store a set of program codes, and the processor 601 is used to call the program codes stored in the memory 603 to execute the terminal posture determination method described in the embodiment of the present application.

[0184] It should be understood that the server 60 is only an example provided in the embodiment of the present application, and the server 60 may have more or fewer components than those shown, may combine two or more components, or may have different configurations of components.

[0185] See also Fig.16 , Fig.16 70 is a schematic diagram of a structure of a device 70 provided in an embodiment of the present application. The device 70 includes a first acquisition module 701, a second acquisition module 702, and a posture determination module 703. In some embodiments, the first acquisition module 701, the second acquisition module 702, and the posture determination module 703 can be operated at Fig.14 The processor 101 or Fig.15 In some embodiments, the first acquisition module 701 may be Figure 2 In the database module 5, the second acquisition module 702 may be Figure 2 The building outline extraction module 3 in the figure, the posture determination module 703 can be Figure 2 6DOF determination module 4 in.

[0186] The first acquisition module 701 is used to acquire multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature, wherein the simulated building outline feature is obtained based on satellite images;

[0187] The second acquisition module 702 is used to obtain the outline features of the building in the picture taken by the terminal;

[0188] The posture determination module 703 is used to determine the simulated posture information corresponding to the simulated building outline feature that matches the building outline feature among the multiple simulated building outline features as the posture information of the terminal.

[0189] The functional modules of the device 70 can be used to implement the following Figure 3 The relevant method steps shown, and Figure 4 , Figure 7 , Fig.12 For the sake of brevity of the specification, the related method steps shown are not repeated here.

[0190] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions, and when the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a network site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, microwave, etc.) mode to another network site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or it can be a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape, etc.), an optical medium (e.g., a DVD, etc.), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0191] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Claims

1. A method for determining a terminal posture, characterized in that: The method comprises: Acquire multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature, wherein the simulated building outline feature is obtained based on satellite images; Obtain building outline features in the image taken by the terminal; Determining the simulated posture information corresponding to the simulated building outline feature that matches the building outline feature among the multiple simulated building outline features as the posture information of the terminal; The obtaining of multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature includes: Acquire a three-dimensional model, wherein the three-dimensional model is obtained based on a plurality of satellite images, and the three-dimensional model includes a plurality of simulated buildings; Determining a road layer in the three-dimensional model; Determining a plurality of sampling points in the road layer; At each sampling point of the plurality of sampling points, projection encoding is performed on the plurality of simulated buildings according to a plurality of simulated posture information to obtain contour features of the plurality of simulated buildings; each contour feature of the simulated building corresponds to a type of simulated posture information; The simulated posture information is the posture information of the sampling point.

2. The method according to claim 1, characterized in that Before determining the simulated posture information corresponding to the simulated building outline feature matching the building outline feature among the multiple simulated building outline features as the posture information of the terminal, the method further includes: The building outline feature is matched with the multiple simulated building outline features to obtain a simulated building outline feature that matches the building outline feature among the multiple simulated building outline features.

3. The method according to claim 2, characterized in that Before matching the building outline feature with the plurality of simulated building outline features, the method further includes: Acquire initial position information and initial posture information of the terminal, where the initial position information and the initial posture information are acquired through a sensor in the terminal; Accordingly, the step of matching the building outline feature with the multiple simulated building outline features to obtain a simulated building outline feature among the multiple simulated building outline features that matches the building outline feature includes: Selecting a portion of simulated building outline features from a plurality of simulated building outline features according to the initial position information and initial posture information of the terminal; The building outline feature is matched with the part of simulated building outline features; the simulated building outline feature with the highest matching degree among the part of simulated building outline features is used as the simulated building outline feature that matches the building outline feature.

4. The method according to claim 3, characterized in that The matching of the building outline feature with the part of the simulated building outline feature comprises: Calculate the initial position information and initial posture information of the building outline feature and the similarity between the model position information and simulated posture information corresponding to each simulated building outline feature in the part of simulated building outline features; the simulated building outline feature with the greatest similarity is used as the simulated building outline feature with the highest matching degree.

5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining the outline features of the building in the picture taken by the terminal includes: Feature extraction is performed on the picture taken by the terminal to obtain the outline features of the building in the picture.

6. The method according to claim 5, characterized in that The extracting features of the picture taken by the terminal to obtain the outline features of the building in the picture includes: Performing morphological processing on the image to eliminate the occlusion of the building outline features by non-target objects, thereby obtaining a processed image; Building semantics are extracted from the processed image to obtain the building outline features.

7. The method according to any one of claims 1 to 4 and 6, characterized in that The obtaining of multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature includes: The multiple simulated building outline features and the simulated posture information corresponding to each simulated building outline feature are received from the server.

8. A device for determining terminal posture, characterized in that: include: A first acquisition module is used to acquire multiple simulated building outline features and simulated posture information corresponding to each simulated building outline feature, wherein the simulated building outline feature is obtained based on satellite images; A second acquisition module is used to acquire the outline features of the building in the picture taken by the terminal; A posture determination module, used to determine the simulated posture information corresponding to the simulated building outline feature matching the building outline feature among the multiple simulated building outline features as the posture information of the terminal; The first acquisition module is specifically used for: Acquire a three-dimensional model, wherein the three-dimensional model is obtained based on a plurality of satellite images, and the three-dimensional model includes a plurality of simulated buildings; Determining a road layer in the three-dimensional model; Determining a plurality of sampling points in the road layer; At each sampling point of the plurality of sampling points, projection encoding is performed on the plurality of simulated buildings according to a plurality of simulated posture information to obtain contour features of the plurality of simulated buildings; each contour feature of the simulated building corresponds to a type of simulated posture information; The simulated posture information is the posture information of the sampling point.

9. The device according to claim 8, characterized in that The posture determination module is used for: The building outline feature is matched with the multiple simulated building outline features to obtain a simulated building outline feature that matches the building outline feature among the multiple simulated building outline features.

10. The device according to claim 9, characterized in that The posture determination module is also used for: Acquire initial position information and initial posture information of the terminal, where the initial position information and the initial posture information are acquired through a sensor in the terminal; Selecting a portion of simulated building outline features from a plurality of simulated building outline features according to the initial position information and initial posture information of the terminal; The building outline feature is matched with the part of simulated building outline features; the simulated building outline feature with the highest matching degree among the part of simulated building outline features is used as the simulated building outline feature that matches the building outline feature.

11. The device according to claim 10, characterized in that The posture determination module is used for: Calculate the initial position information and initial posture information of the building outline feature and the similarity between the model position information and simulated posture information corresponding to each simulated building outline feature in the part of simulated building outline features; the simulated building outline feature with the greatest similarity is used as the simulated building outline feature with the highest matching degree.

12. The device according to any one of claims 8 to 11, characterized in that: The second acquisition module is used for: Feature extraction is performed on the picture taken by the terminal to obtain the outline features of the building in the picture.

13. The device according to claim 12, characterized in that The second acquisition module is used for: Performing morphological processing on the image to eliminate the occlusion of the building outline features by non-target objects, thereby obtaining a processed image; Building semantics are extracted from the processed image to obtain the building outline features.

14. The device according to any one of claims 8 to 11 and 13, characterized in that The first acquisition module is used for: The multiple simulated building outline features and the simulated posture information corresponding to each simulated building outline feature are received from the server.

Images