Positioning method and device, electronic equipment and computer readable storage medium
By generating and using the conversion matrix under the visual inertial odometer coordinate system, the real positioning error problems caused by transmission delay and sensor accuracy are solved, and the accuracy and user experience of augmented reality indoor navigation are improved.
Patent Information
- Application Number
- CN202310638131.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
Among the existing augmented reality indoor navigation technology, the positioning error is large, mainly due to the transmission delay and low sensor accuracy, resulting in poor navigation effects.
By obtaining the visual image frame of the terminal, the virtual position information under the visual inertial odometer (VIO) coordinate system is calculated, and the transformation matrix is generated, which is directly used to determine the real position of the terminal, avoiding relying on the server-side positioning engine and sensors.
Improve the accuracy of positioning, reduce the impact of transmission delay and sensor accuracy on positioning, and enhance the accuracy of navigation and user experience.
Smart Images

Figure CN119071897A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of the Internet, and relate to but are not limited to a positioning method, device, electronic device, and computer-readable storage medium. Background Art
[0002] At present, in the field of augmented reality (AR) indoor navigation, in the positioning process, the geographical location is usually determined by the comprehensive positioning engine on the server side, and the direction angle information is realized by mobile phone sensors (such as compass).
[0003] However, when positioning through the integrated positioning engine on the server side, the impact of transmission delay on positioning accuracy is not considered. In addition, when obtaining angular information through the mobile phone sensor, the impact of low sensor accuracy on angular information is not considered. In summary, there will be a large positioning error during indoor navigation, resulting in poor navigation results. Summary of the invention
[0004] The embodiments of the present application provide a positioning method, device, electronic device and computer-readable storage medium, which can at least be applied to the field of maps and map positioning, can effectively improve the problem of large positioning errors caused by low positioning sensor accuracy and positioning delay, and improve the accuracy of positioning.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] An embodiment of the present application provides a positioning method, including: in response to the positioning request, obtaining the first posture information of the terminal; the positioning request includes the visual image frame at the current moment collected by the terminal; the first posture information is the real posture information in the geographic coordinate system corresponding to the visual image frame at the current moment; obtaining the first virtual posture information in the visual inertial odometer (VIO, Visual Inertial Odometry) coordinate system corresponding to the visual image frame at the current moment; using the first posture information and the first virtual posture information, generating a transformation matrix; using the transformation matrix to transform the second virtual posture information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, to obtain the second virtual posture information after the transformation; the second virtual posture information after the transformation is used to determine the real posture information of the terminal at the next moment.
[0007] An embodiment of the present application provides a positioning device, which includes: a first acquisition module, used to obtain the first posture information of the terminal in response to a positioning request; the positioning request includes a visual image frame at the current moment collected by the terminal; the first posture information is the real posture information in the geographic coordinate system corresponding to the visual image frame at the current moment; a second acquisition module, used to obtain the first virtual posture information in the VIO coordinate system corresponding to the visual image frame at the current moment; a generation module, used to generate a transformation matrix using the first posture information and the first virtual posture information; a processing module, used to use the transformation matrix to transform the second virtual posture information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, to obtain the second virtual posture information after the transformation; the second virtual posture information after the transformation is used to determine the real posture information of the terminal at the next moment.
[0008] In some embodiments, the device also includes: a conversion module, used to obtain first reference information of a geographic coordinate system; the first reference information includes reference orientation information and reference position information of the geographic coordinate system; based on the first reference information, the first posture information is converted into second posture information; the second posture information is the real posture information in the Cartesian coordinate system corresponding to the visual image frame at the current moment; the processing module is also used to generate the transformation matrix using the second posture information and the first virtual posture.
[0009] In some embodiments, the conversion module is also used to: perform plane projection processing on the geographic coordinate system based on the first reference information to obtain the geographic coordinate system after the plane projection processing; obtain second reference information of the geographic coordinate system after the plane projection processing; the second reference information includes the reference orientation information and reference position information of the geographic coordinate system after the plane projection processing; by adjusting the second reference information, the geographic coordinate system after the plane projection processing is converted into a Cartesian coordinate system; obtain third reference information of the Cartesian coordinate system; the third reference information includes the reference orientation information and reference position information of the Cartesian coordinate system; based on the third reference information, convert the first posture information into second posture information.
[0010] In some embodiments, the device also includes: a verification module, used to obtain third posture information of the terminal; the third posture information is the real posture information under the geographic coordinates corresponding to the visual image frame of the previous moment before the current moment and adjacent to the current moment; the third posture information and the first posture information are verified to obtain a verification result; the generation module is also used to: when the verification result is a successful verification, use the first posture information and the first virtual posture information to generate the transformation matrix.
[0011] In some embodiments, the verification module is also used to: determine the real posture deviation using the first posture information and the third posture information; obtain the third virtual posture information in the VIO coordinate system corresponding to the visual image frame of the previous moment before the current moment and adjacent to the current moment; determine the virtual posture deviation using the first virtual posture information and the third virtual posture information; and when the real posture deviation matches the virtual posture deviation, determine that the verification result is a successful verification.
[0012] In some embodiments, the device also includes: a path generation module, which is used to receive the target position information of this positioning sent by the terminal; determine the path information based on the second virtual posture information after the conversion processing and the target position information; the path information includes guidance information for reaching the target position corresponding to the target position information.
[0013] In some embodiments, the path generation module is also used to: obtain model information corresponding to the visual image frame at the current moment; generate a virtual scene in the VIO coordinate system based on the model information corresponding to the visual image frame at the current moment; determine guidance information for reaching the target position from the position corresponding to the first posture information based on the first posture information and the target position information; fuse the guidance information for reaching the target position from the position corresponding to the first posture information into the virtual scene through the transformation matrix to obtain a fused virtual scene; and determine guidance information for reaching the target position corresponding to the target position information from the fused virtual scene based on the second virtual posture information after the conversion processing.
[0014] In some embodiments, the path generation module is also used to: obtain model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment; when the model information corresponding to the visual image frame at the current moment is different from the model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, based on the model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, update the visual posture information corresponding to the fused virtual scene to obtain an updated virtual scene; based on the second virtual posture information after the conversion processing, determine the guidance information for reaching the target position corresponding to the target position information from the updated virtual scene.
[0015] In some embodiments, the positioning request also includes auxiliary feature information of the real position corresponding to the visual image frame at the current moment; the auxiliary feature information is used to distinguish similar positions; the first acquisition module is also used to: respond to the positioning request, determine the auxiliary position information corresponding to the visual image frame at the current moment based on the auxiliary feature information; perform feature matching on the visual image frame at the current moment with a preset three-dimensional point cloud map to obtain a matching result; determine the first posture information based on the auxiliary position information and the matching result.
[0016] An embodiment of the present application provides an electronic device, including: a memory, used to store executable instructions; and a processor, used to implement the above-mentioned positioning method when executing the executable instructions stored in the memory.
[0017] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein a processor of an electronic device reads the executable instructions from the computer-readable storage medium and implements the above-mentioned positioning method when executing the executable instructions.
[0018] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned positioning method.
[0019] The embodiment of the present application has the following beneficial effects: in response to the positioning request, after obtaining the first position information of the terminal, the first virtual position information in the VIO coordinate system corresponding to the visual image frame at the current moment is obtained, and the first position information and the first virtual position information are used to generate a conversion matrix; the second virtual position information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment is converted by using the conversion matrix to obtain the second virtual position information after the conversion. After obtaining the conversion matrix through the real positioning information of the server at the current moment and the VIO virtual positioning information, in the positioning process, the conversion matrix is directly used to convert the VIO virtual position information at the next moment, so as to obtain the real position information of the terminal at the next moment, without relying on the server-side positioning engine and the terminal sensor to locate the next moment, avoiding the influence of transmission delay and sensor accuracy on the positioning accuracy, and improving the positioning accuracy. In addition, since the conversion matrix at the next moment is determined based on the current moment, the server positioning can be performed intermittently during the positioning process to update the conversion matrix, further improving the positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 is an optional architecture diagram of a positioning system provided in an embodiment of the present application;
[0021] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0022] Figure 3 This is an optional flowchart of the positioning method provided in the embodiment of the present application;
[0023] Figure 4 is another optional flowchart of the positioning method provided in the embodiment of the present application;
[0024] Figure 5 It is a schematic diagram of the implementation process of determining the first posture information provided by an embodiment of the present application;
[0025] Figure 6 This is a schematic diagram of an implementation flow of generating a conversion matrix provided in an embodiment of the present application;
[0026] Figure 7 It is a schematic diagram of the implementation process of the positioning method provided in the embodiment of the present application;
[0027] Figure 8 This is a schematic diagram of the implementation process of generating a positioning request provided in an embodiment of the present application;
[0028] Fig. 9 It is a schematic diagram of the architecture of the world coordinate system and the VIO coordinate system provided in the embodiment of the present application;
[0029] Fig.10 This is another schematic diagram of a process flow for implementing the generation of a conversion matrix provided in an embodiment of the present application;
[0030] Fig.11 It is a schematic diagram of the scene rendering process provided by an embodiment of the present application;
[0031] Fig.12 It is a schematic diagram of the positioning process provided by the embodiment of the present application;
[0032] Fig.13 is a schematic diagram of a positioning case of the positioning method provided in an embodiment of the present application;
[0033] Fig.14 It is a flowchart of a positioning method with scene rendering provided in an embodiment of the present application. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0035] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.
[0036] Before explaining the positioning method of the embodiment of the present application, the professional terms involved in the embodiment of the present application are first explained.
[0037] (1) WeChat Mini Program: It is an application that can be used without downloading. Users can open the application by scanning or searching, making the application within reach.
[0038] (2) Indoor navigation: This is a technology that uses AR paths and virtual arrows in indoor spaces, combining AR virtual arrows and directions with indoor maps to guide users to their destinations. In this way, users in large buildings can still use precise positioning functions to determine their location and find their destination even inside the building.
[0039] (3) Indoor visual positioning: It is the process of using laser visual map acquisition equipment to collect the existing indoor environment, establish a three-dimensional point cloud map, and predict the current position and posture of the camera in the indoor environment through a single image taken by the camera.
[0040] (4) AR: It is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time positioning and registration, intelligent interaction, and sensing to simulate computer-generated text, images, three-dimensional models, music, videos and other virtual information and apply them to the real world. The two types of information complement each other, thus achieving an enhancement of the real world.
[0041] (5) VIO: It is an algorithm that integrates camera images and inertial measurement unit (IMU) data to achieve simultaneous localization and mapping (SLAM). It determines the position and direction by analyzing relevant camera images; VIO under WeChat Mini Program refers to the real open under the Mini Program visual application programming interface (API).
[0042] (6) SLAM: A technology that instructs an autonomous mobile robot to start from an unknown location in an unknown environment, locate its own position and posture by repeatedly observing map features (such as corners and pillars) during movement, and then incrementally construct a map based on its own position, thereby achieving the purposes of positioning and map construction at the same time.
[0043] (7) IMU: A device that measures the three-axis attitude angle (or angular rate) and acceleration of an object, such as the gyroscope and accelerometer on a mobile phone.
[0044] (8) Structure-From-Motion (SFM): It is an offline algorithm for 3D reconstruction based on collected unordered images.
[0045] (9) Geographic Coordinate System (GCS): A coordinate system that uses a three-dimensional spherical surface to define locations on the earth's surface, so as to reference points on the earth's surface through longitude and latitude, such as the GCJ02 coordinate system.
[0046] In the relevant technology, the main process of the indoor AR navigation solution under the mini program includes: obtaining Bluetooth, WiFi, accelerometer, compass and gyroscope information through the interface provided by the mini program or APP; obtaining real-time location information through the integrated positioning engine, and updating the results every second; obtaining the posture information and steering information of the mobile phone; using the real-time positioning results to obtain the surrounding POI, signboards and road information through the integrated map engine; obtaining camera photos through the program or APP, and making basic judgments on the mobile phone; sending photos to the server for image recognition when conditions are met; among them, when stores and logos appear in the camera photos, and the picture changes significantly If the system detects changes in the location and the periodic interval time, it determines that the conditions are met; the system confirms or corrects the current position and direction based on the store name and logo; confirms whether the user is on the road based on road recognition, and then determines the specific walking road based on the location, and finally confirms the walking direction on the current road based on the compass direction; confirms the relative position of the road in the photo based on the road photo and the phone posture; overlays the AR icon on the photo; overlays POI, logo and other information on the camera in proportion to the front, back, left and right relative positions; when the road turns, the turning state is confirmed by the gyroscope; when turning, the AR icon changes with the turn.
[0047] From the above description, it can be seen that in terms of positioning implementation, the relevant technology relies on the server-side integrated positioning engine and mobile phone sensors for positioning, that is, the mobile phone compass. In this way, there will be the following two problems. On the one hand, when positioning is performed through the server-side integrated positioning engine, the influence of the request transmission delay is not taken into account. In areas with poor network, there is a problem of untimely positioning. At the same time, in the process of image recognition based on the mobile phone image by the server-side integrated positioning engine, there may be a transmission delay of the recognition result, resulting in a large positioning error; on the other hand, when positioning is performed through the mobile phone compass, electromagnetic interference or compass frame rate in the indoor environment will affect the positioning accuracy. In summary, in the positioning process, due to the transmission delay or sensor accuracy problem, the positioning error is large, and there is a problem of low positioning accuracy, which will affect the navigation effect.
[0048] The positioning method provided in the embodiment of the present application, first, in response to the positioning request, after obtaining the first posture information of the terminal, obtains the first virtual posture information in the VIO coordinate system corresponding to the visual image frame at the current moment; then, using the first posture information and the first virtual posture information, a conversion matrix is generated; finally, the second virtual posture information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment is converted using the conversion matrix to obtain the second virtual posture information after the conversion processing. After obtaining the conversion matrix through the server positioning information and the VIO virtual positioning information at the current moment, in the positioning process, the VIO virtual posture information (including position information and orientation) at the next moment is directly converted using the conversion matrix, so as to obtain the real posture information at the next moment, without relying on the server-side positioning engine and the terminal sensor to position each moment, avoiding the influence of transmission delay and sensor accuracy on the positioning accuracy, and improving the positioning accuracy. In addition, since the conversion matrix at the next moment is determined based on the current moment, the server positioning can be performed intermittently during the positioning process to update the conversion matrix, further improving the positioning accuracy.
[0049] Here, firstly, an exemplary application of the positioning device of the embodiment of the present application is described, and the positioning device is an electronic device for implementing the positioning method. In one implementation, the positioning device (i.e., electronic device) provided in the embodiment of the present application can be implemented as a terminal or a server. In one implementation, the positioning device provided in the embodiment of the present application can be implemented as a laptop, a tablet computer, a desktop computer, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, an intelligent robot, an intelligent home appliance, and an intelligent vehicle-mounted device, etc., any terminal with video rendering and video display functions; in another implementation, the positioning device provided in the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application. Next, an exemplary application when the positioning device is implemented as a server will be described.
[0050] See also Figure 1 , Figure 1 It is an optional architectural diagram of the positioning system provided in the embodiment of the present application. The embodiment of the present application is explained by taking the positioning method applied to any positioning application as an example. In the embodiment of the present application, the positioning system 10 includes at least a terminal 100, a network 200 and a server 300. Among them, a positioning application can be installed on the terminal 100, and the user can send a positioning request on the client of the positioning application; the server 300 can be a server of the positioning application, and the server 300 can constitute the positioning device of the embodiment of the present application. The terminal 100 is connected to the server 300 via the network 200, and the network 200 can be a wide area network or a local area network, or a combination of the two.
[0051] In the embodiment of the present application, a positioning application is run on the terminal 100. When performing positioning processing, the terminal 100 receives the visual image frame at the current moment sent by the user, and encapsulates the visual image frame at the current moment into a positioning request, and sends the positioning request to the server 300 through the network 200. In response to the positioning request, the server 300 obtains the first pose information of the terminal, and obtains the first virtual pose information in the VIO coordinate system corresponding to the visual image frame at the current moment from the terminal 100; then, using the first pose information and the first virtual pose information, a conversion matrix is generated; then, using the conversion matrix, the second virtual pose information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment is converted to obtain the converted second virtual pose information. After determining the converted second virtual pose information, the server 300 can send the converted second virtual pose information to the terminal 100 through the network 200, and the terminal 100 can display the converted second virtual pose information on the current interface.
[0052] In other embodiments, the positioning device can also be implemented as a terminal, that is, a positioning application is installed on the terminal 100, and the positioning method of the embodiment of the present application can be implemented with the terminal as the execution subject. In the implementation process, the terminal 100 obtains the visual image frame collected by the user at the current moment through the positioning application, and in response to the positioning request, obtains the first position information of the terminal from the server 300 through the network 200; then, the terminal 100 obtains the first virtual position information in the VIO coordinate system corresponding to the visual image frame at the current moment, and uses the first position information and the first virtual position information to generate a transformation matrix; then, using the transformation matrix, the second virtual position information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment is transformed and processed to obtain the second virtual position information after the transformation. After obtaining the second virtual position information after the transformation, the terminal 100 can display the obtained second virtual position information after the transformation on the current interface.
[0053] The positioning method provided in the embodiment of the present application can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 can be a cloud server. The cloud server generates a conversion matrix, or uses the conversion matrix to convert the second virtual posture information in the VIO coordinate system corresponding to the visual image frame of the next moment after the current moment and adjacent to the current moment, to obtain the converted second virtual posture information.
[0054] In some embodiments, a cloud storage may be provided, and the first pose information, the first virtual pose information and other information may be stored in the cloud storage, or the conversion matrix may be stored in the cloud storage. In this way, when the second virtual pose information is subsequently converted, the stored conversion matrix may be selected from the cloud storage for conversion, thereby improving the efficiency of positioning.
[0055] It should be noted here that cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can be achieved through cloud computing.
[0056] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device shown may be a positioning device, which includes: at least one processor 310, a memory 350, at least one network interface 320 and a user interface 330. The various components in the positioning device are coupled together through a bus system 340. It is understood that the bus system 340 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 340 is not described in detail. Figure 2 Various buses are labeled as bus system 340 .
[0057] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0058] The user interface 330 includes one or more output devices 331 that enable presentation of media content, and one or more input devices 332 .
[0059] The memory 350 may be removable, non-removable or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drive, optical disk drive, etc. The memory 350 may optionally include one or more storage devices physically away from the processor 310. The memory 350 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 350 described in the embodiment of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 can store data to support various operations, and examples of these data include programs, modules, and data structures or subsets or supersets thereof, as exemplarily described below.
[0060] An operating system 351 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic businesses and processing hardware-based tasks; a network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320. Exemplary network interfaces 320 include: Bluetooth, wireless compatibility certification (WiFi), and Universal Serial Bus (USB), etc.; an input processing module 353 is used to detect one or more user inputs or interactions from one of the one or more input devices 332 and translate the detected inputs or interactions.
[0061] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A positioning device 354 stored in the memory 350 is shown, and the positioning device 354 may be a positioning device in an electronic device, and may be software in the form of a program or a plug-in, and includes the following software modules: a receiving module 3541, a first acquisition module 3542, a second acquisition module 3543, a generation module 3544, a processing module 3545, and a path generation module 3456. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.
[0062] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the positioning method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.
[0063] The positioning method provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the positioning method provided in each embodiment of the present application can be executed by a server, can be executed by a terminal, or can be executed through interaction between a server and a terminal.
[0064] Figure 3 This is an optional flow chart of the positioning method provided in the embodiment of the present application. Figure 3 The steps shown are explained, and it should be noted that Figure 3 The positioning method in the example is explained by taking the server as the execution subject. Figure 3 As shown, the method includes the following steps S101 to S106:
[0065] Step S101, in response to a positioning request, obtaining the first posture information of the terminal; the positioning request includes a visual image frame at the current moment collected by the terminal; the first posture information is the real posture information in the geographic coordinate system corresponding to the visual image frame at the current moment.
[0066] Here, the first posture information at least includes the geographical location information and orientation information in the geographical coordinate system corresponding to the visual image frame at the current moment.
[0067] In an embodiment of the present application, the visual image frame at the current moment may be the visual image frame at the current moment captured by the terminal when the user is in a different environment, such as the visual image frame at the current moment captured by the terminal when the user is in an indoor building, or the visual image frame at the current moment captured by the terminal when the user is in a scenic spot. The terminal may capture the visual image frame at the current moment when the user is in different environments through a camera, and render the captured visual image frame at the current moment to display the rendered visual image frame at the current moment on the current interface of the terminal. For example, the terminal may perform Canvas rendering on the captured visual image frame at the current moment, and display the Canvas rendered visual image frame at the current moment on the current interface of the terminal.
[0068] In an embodiment of the present application, a positioning application can be run on the terminal, specifically, the positioning application can be run in the form of a small program, or the positioning application can be run in the form of an H5 page. The visual image frame at the current moment can be a visual image frame at the current moment collected by the user through the camera operation, or it can be a visual image frame at the current moment automatically collected by the positioning application according to preset conditions. Then, the terminal can encapsulate the visual image frame at the current moment into a positioning request and send it to the server.
[0069] For the first position information, the server can use other electronic devices, such as robots, to collect images of the environment that needs to be positioned in advance; through SFM technology, a three-dimensional image point cloud map is established based on the collected images, so that the server can subsequently respond to the positioning request and obtain the first position information of the terminal based on the three-dimensional image point cloud map.
[0070] In the embodiment of the present application, after the server receives the positioning request sent by the terminal, in response to the positioning request, the visual image frame at the current moment can be feature matched with the three-dimensional image point cloud map to obtain the geographical location information and orientation information in the geographical coordinate system corresponding to the visual image frame at the current moment. For example, the geographical coordinate system may include the GCJ02 coordinate system, and the server can obtain the geographical location information and orientation information in the GCJ02 coordinate system corresponding to the visual image frame at the current moment.
[0071] In an embodiment of the present application, the north orientation angle can be used as the reference orientation of the geographic coordinate system, and a specific position can be used as the reference position of the geographic coordinate system. That is to say, the server can obtain the orientation information of the visual image frame at the current moment in the GCJ02 coordinate system relative to the reference orientation, as well as the geographic location relative to the reference position, such as the longitude and latitude relative to the reference position.
[0072] In the process of realizing positioning, the server is relied on to determine the geographic location information and orientation information of the visual image frame at the current moment in the geographic coordinate system, without relying on the terminal sensor to determine the orientation information. This can avoid the problem of inaccurate orientation information caused by electromagnetic interference of the sensor or insufficient frame rate, thereby improving the accuracy of positioning.
[0073] Step S102, obtaining first virtual pose information in the VIO coordinate system corresponding to the visual image frame at the current moment.
[0074] Here, the first virtual posture information at least includes virtual position information and orientation information in the VIO coordinate system corresponding to the visual image frame at the current moment.
[0075] It should be noted that the first virtual posture information is relative position information, that is, the position information and orientation information of the visual image frame at the current moment relative to the reference posture of the VIO coordinate system. The reference posture of the VIO coordinate system can be dynamically changed, and the reference posture of the VIO coordinate system is associated with the position and orientation of the terminal when the server establishes the VIO coordinate system. When the user triggers the VIO function on the terminal, the terminal can obtain the position information and orientation information at the moment when the VIO function is triggered, and establish the VIO coordinate system with the position information and orientation information of the terminal at the moment when the VIO function is triggered as the reference information; then, the terminal determines the first virtual posture information in the VIO coordinate system corresponding to the visual image frame at the current moment based on the reference information of the VIO coordinate system, and sends the first virtual posture information to the server.
[0076] In an embodiment of the present application, the VIO coordinate system may use a rectangular coordinate system to define virtual pose information. For example, when the VIO coordinate system uses a three-dimensional rectangular coordinate system to define virtual pose information, the first virtual pose information may include virtual position information and orientation information corresponding to the X-axis, Y-axis, and Z-axis.
[0077] In the embodiment of the present application, in the process of determining the first virtual posture information, since the positioning function corresponding to the VIO function has a high accuracy, it can return accurate virtual posture information within a certain interval, thereby improving the positioning accuracy of the virtual posture information.
[0078] Step S103: Generate a transformation matrix using the first posture information and the first virtual posture information.
[0079] Here, the transformation matrix is used to achieve the conversion between the VIO coordinate system and the geographic coordinate system.
[0080] In an embodiment of the present application, the geographic coordinate system defines the real geographic location information and real orientation information corresponding to the visual image frame at the current moment through a spherical surface, and the VIO coordinate system defines the virtual geographic location information and virtual orientation information corresponding to the visual image frame at the current moment through a rectangular coordinate system. Therefore, the server can establish a relationship with the VIO coordinate system based on the reference information of the geographic coordinate system, thereby generating a transformation matrix.
[0081] In an embodiment of the present application, the Cartesian coordinate system may include a rectangular coordinate system and an oblique coordinate system, that is, the Cartesian coordinate system may define the real geographic location information and real orientation information corresponding to the visual image frame at the current moment through the rectangular coordinate system. Therefore, the server may first convert the geographic coordinate system into a Cartesian coordinate system, and then establish the relationship between the Cartesian coordinate system and the VIO coordinate system. Specifically, the server may perform plane projection processing on the geographic coordinate system based on the reference information of the geographic coordinate system to obtain the geographic coordinate system after the plane projection processing; then, adjust the reference information of the geographic coordinate system after the plane projection processing to convert the geographic coordinate system after the plane projection processing into the Cartesian coordinate system; then, based on the reference information of the Cartesian coordinate system, obtain the second posture information in the Cartesian coordinate system corresponding to the visual image frame at the current moment; and determine the conversion matrix using the second posture information in the Cartesian coordinate system and the first virtual posture information in the VIO coordinate system.
[0082] It should be noted that the geographic coordinate system may include the GCJ02 coordinate system. Since the GCJ02 coordinate system is usually a coordinate system obtained after deflection processing according to a deflection algorithm, it is impossible to directly convert the GCJ02 coordinate system into a Cartesian coordinate system. In order to realize the conversion between the geographic coordinate system and the Cartesian coordinate system, the server can convert the GCJ02 coordinate system into the WG584 coordinate system based on the reference information of the GCJ02 coordinate system through a correction algorithm, and perform plane projection processing on the WG584 coordinate system to obtain the WG584 coordinate system after the plane projection processing; then, by adjusting the reference information of the WG584 coordinate system after the plane projection processing, the WG584 coordinate system after the plane projection processing is converted into a Cartesian coordinate system; then, based on the reference information of the Cartesian coordinate system, the second posture information in the Cartesian coordinate system corresponding to the visual image frame at the current moment is obtained, and then the conversion matrix is determined.
[0083] In some embodiments, in the process of determining the conversion matrix, in order to avoid the cumulative error caused by the posture information, the server can re-acquire the first posture information and the first virtual posture information at intervals, so as to update the conversion matrix according to the re-acquired first posture information and the first virtual posture information, and obtain an updated conversion matrix. For example, the server can obtain the first posture information at intervals of the first time length, and the value of the first time length can be set as needed, such as 5 seconds; the server can also obtain the first virtual posture information at intervals of the second time length, and the value of the second time length can be set as needed, such as 1 second. In this way, the real-time positioning in the positioning process can be improved, and the positioning accuracy is guaranteed.
[0084] Step S104, using the transformation matrix to transform the second virtual posture information in the VIO coordinate system corresponding to the visual image frame of the next moment after the current moment and adjacent to the current moment, to obtain the transformed second virtual posture information; the transformed second virtual posture information is used to determine the real posture information of the terminal at the next moment.
[0085] Here, the second virtual posture information includes at least virtual position information and orientation information in the VIO coordinate system corresponding to the visual image frame of the next moment after the current moment and adjacent to the current moment.
[0086] In an embodiment of the present application, a positioning application can be run on the terminal, specifically, the positioning application can be run in the form of a mini program, or in the form of an H5 page. The visual image frame of the next moment after the current moment and adjacent to the current moment can be a visual image frame of the next moment after the current moment and adjacent to the current moment collected by the user through the camera, or it can be a visual image frame of the next moment after the current moment and adjacent to the current moment automatically collected by the positioning application according to preset conditions. After the terminal collects the visual image frame of the next moment after the current moment and adjacent to the current moment, the visual image frame of the next moment after the current moment and adjacent to the current moment can be sent to the server in real time, so that the server obtains the second virtual posture information in the VIO coordinate system corresponding to the visual image frame of the next moment after the current moment and adjacent to the current moment.
[0087] It should be noted that the second virtual posture information is the position information and orientation information of the visual image frame at the next moment after the current moment and adjacent to the current moment relative to the reference information of the VIO coordinate system. In other words, the server obtains the second virtual posture information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment based on the reference information of the VIO coordinate system.
[0088] In an embodiment of the present application, after the server obtains the second virtual posture information, it can use the transformation matrix to transform the second virtual posture information to obtain the transformed second virtual posture information. For example, if the transformation matrix is used to transform the virtual posture information in the VIO coordinate system into the real posture information in the world coordinate system, the server can directly use the transformation matrix to transform the second virtual posture information; if the transformation matrix is used to transform the real posture information in the world coordinate system into the virtual posture information in the VIO coordinate system, the server can use the inverse matrix of the transformation matrix to transform the second virtual posture information.
[0089] In an embodiment of the present application, the server can quickly convert the second virtual posture information through the conversion matrix, so as to determine the real posture information of the terminal at the next moment, that is, to locate the terminal at the next moment, without the need for positioning through feature matching and terminal sensors. In this way, the positioning error can be reduced and the positioning accuracy can be improved.
[0090] In some embodiments, in an application scenario where a user needs path navigation, path information for this path navigation can also be generated for the user. While positioning the user, the path information can also be displayed to the user to guide the user to the destination corresponding to this path navigation. After obtaining the target location information of this path navigation sent by the terminal, the server determines the path information based on the converted second virtual posture information and the target location information; the path information includes guidance information for reaching the target location corresponding to the target location information.
[0091] Here, the guidance information at least includes an identification of the target location corresponding to the target location information, route guidance, and point of interest (POI) information.
[0092] After generating the guidance information, the server can send the guidance information to the terminal; the terminal displays the guidance information in the client of the positioning application to achieve the superposition of the guidance information and the real visual image frame. In this way, the superposition accuracy of the virtual path information and the real image during the positioning process can be effectively improved, so that the user's navigation immersion is better.
[0093] The positioning method provided in the embodiment of the present application, in response to the positioning request, obtains the first position information of the terminal; obtains the first virtual position information in the VIO coordinate system corresponding to the visual image frame at the current moment, and generates a conversion matrix using the first position information and the first virtual position information; uses the conversion matrix to convert the second virtual position information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, and obtains the second virtual position information after the conversion. After obtaining the conversion matrix through the real positioning information and VIO virtual position information of the server at the current moment, in the positioning process, the conversion matrix is directly used to convert the VIO virtual position information at the next moment, so as to obtain the real position information at the next moment, without relying on the server-side positioning engine and the terminal sensor for the next moment, avoiding the influence of transmission delay and sensor accuracy on the positioning accuracy, and improving the positioning accuracy. In addition, since the conversion matrix at the next moment is determined based on the current moment, the server positioning can be performed intermittently during the positioning process to update the conversion matrix, further improving the positioning accuracy.
[0094] In some embodiments, the positioning system includes at least a terminal and a server, and a positioning application is installed on the terminal, and the positioning application can provide a positioning function. The server is a server of the positioning application, and the positioning method of each embodiment of the present application can be executed by interaction between the server of the positioning application and the terminal.
[0095] Figure 4 is another optional flow chart of the positioning method provided in the embodiment of the present application, wherein: Figure 4 The implementation process of the positioning method implemented by the interaction between the terminal and the server is shown, including the following steps S201 to S214:
[0096] Step S201: The terminal collects a visual image frame at the current moment.
[0097] Here, the terminal may collect the visual image frame at the current moment through the camera. The visual image frame at the current moment may be automatically collected by the positioning application according to preset conditions, or may be collected by the user by clicking a button on the client of the positioning application.
[0098] After collecting the visual image frame at the current moment, the terminal can also render the visual image frame at the current moment to display the rendered visual image frame at the current moment on the current interface of the positioning application for the user to view. For example, the terminal can perform Canvas rendering on the visual image frame at the current moment.
[0099] Step S202: The terminal encapsulates the collected visual image frame at the current moment into a positioning request.
[0100] Here, the positioning request is used to request positioning of the user's current location.
[0101] The terminal may convert the visual image frame at the current moment after the rendering process into an image in a specific format, so as to encapsulate the image in the specific format into the positioning request. For example, the terminal may convert the visual image frame at the current moment after the rendering process into an image in a JPG format.
[0102] Step S203: The terminal sends a positioning request to the server.
[0103] Step S204: The server obtains the first position information of the terminal in response to the positioning request.
[0104] Here, in the process of the server acquiring the first pose information, since there may be similar positions such as floors or elevator entrances, the server may make an incorrect recognition, resulting in inaccurate first pose information. In this regard, in an embodiment of the present application, the terminal can also collect auxiliary feature information of the real position corresponding to the visual image frame at the current moment, and send it to the server through a positioning request, so that the server determines the first pose information based on the auxiliary feature information; that is, the positioning request can also include the auxiliary feature information of the real position corresponding to the visual image frame at the current moment, and the auxiliary feature information is used to distinguish similar positions.
[0105] In some embodiments, see Figure 5 , Figure 5 It is shown that obtaining the first pose information in step S204 can be achieved by following steps S2041 to S2043:
[0106] Step S2041, in response to the positioning request, determining the auxiliary position information corresponding to the visual image frame at the current moment based on the auxiliary feature information.
[0107] Here, the auxiliary feature information may include at least one of the following: Bluetooth information; WiFi fingerprint information; the auxiliary location information may include information such as floor, elevator entrance, etc.
[0108] When determining the auxiliary location information, different locations correspond to different auxiliary feature information, and the server can determine the auxiliary location information corresponding to the visual image frame at the current moment based on the auxiliary feature information. For example, when the auxiliary feature information includes the first Bluetooth information, the server can determine that the visual image frame at the current moment corresponds to the first floor; when the auxiliary feature information includes the second Bluetooth information, the server can determine that the visual image frame at the current moment corresponds to the third floor.
[0109] Step S2042, feature matching is performed on the visual image frame at the current moment with the preset three-dimensional point cloud map to obtain a matching result.
[0110] Here, the server extracts features from the visual image frame at the current moment, matches the extracted features with the preset three-dimensional point cloud map, and obtains a matching result; the matching result includes at least one visual image frame that matches the extracted features, and each visual image frame corresponds to a pose information.
[0111] Step S2043, determining the first pose information based on the auxiliary position information and the matching result.
[0112] Here, the server can determine the first position information corresponding to the visual image frame at the current moment by comparing the auxiliary position information with the position information corresponding to at least one visual image frame matching the extracted features.
[0113] In the embodiment of the present application, the server can effectively improve the problem of low recognition accuracy of the first pose information in similar scenes by using the auxiliary position information as reference information, thereby improving the positioning accuracy.
[0114] Step S205: The server obtains first virtual pose information in the VIO coordinate system corresponding to the visual image frame at the current moment.
[0115] Here, the server can determine the reference pose of the VIO coordinate system and establish the VIO coordinate system based on the VIO function of the visual API when the user triggers the VIO function; then, for the visual image frame at the current moment, the server obtains the first virtual pose information relative to the reference pose in the VIO coordinate system based on the VIO function.
[0116] Step S206: The server obtains the third posture information of the terminal.
[0117] Here, the third posture information is the real posture information under the geographical coordinates corresponding to the visual image frame at the previous moment before the current moment and adjacent to the current moment;
[0118] In the process of determining the first pose information, the server may also encounter a large deflection error between the first pose information and the actual pose information. Therefore, in some embodiments, a secondary verification mechanism can be introduced to perform mutual verification between the visual image frame at the previous moment before the current moment and adjacent to the current moment and the image frame at the current moment to improve the accuracy of the first pose information.
[0119] Here, after the server obtains the visual image frame of the previous moment before the current moment and adjacent to the current moment, it can determine the third posture information based on the preset three-dimensional point cloud map and the auxiliary position information corresponding to the visual image frame of the previous moment before the current moment and adjacent to the current moment.
[0120] In some embodiments, if the server cannot obtain the third posture information of the terminal, that is, the positioning fails, the server can obtain the posture information of the visual image frame at the next moment after the current moment and adjacent to the current moment, and use the image frame at the current moment and the visual image frame at the next moment after the current moment and adjacent to the current moment for mutual verification.
[0121] Step S207: The server uses the first pose information and the third pose information to determine the actual pose deviation.
[0122] Here, the real posture deviation includes the real relative position and real relative orientation in the world coordinate system between the visual image frame at the previous moment before the current moment and adjacent to the current moment and the visual image frame at the current moment.
[0123] Step S208: The server obtains the third virtual pose information in the VIO coordinate system corresponding to the visual image frame of the previous moment before the current moment and adjacent to the current moment.
[0124] For the visual image frame of the previous moment before the current moment and adjacent to the current moment, the server can obtain the third virtual posture information relative to the reference posture in the VIO coordinate system based on the VIO function.
[0125] Step S209: The server determines a virtual posture deviation by using the first virtual posture information and the third virtual posture information.
[0126] Here, the virtual posture deviation includes the virtual relative position and virtual relative orientation in the VIO coordinate system between the visual image frame at the previous moment before the current moment and adjacent to the current moment and the visual image frame at the current moment.
[0127] Step S210: When the real posture deviation matches the virtual posture deviation, the server determines that the verification result is successful.
[0128] In some embodiments, the server may compare the real posture deviation with the virtual posture deviation, and determine that the verification result is successful when the real posture deviation is consistent with the virtual posture deviation.
[0129] In other embodiments, when the real posture deviation is inconsistent with the virtual posture deviation, the server can determine that the verification result is a verification failure, that is, there is a positioning error or the VIO coordinate system is offset. In this case, the server can obtain the visual image frame of the next moment after the current moment and adjacent to the current moment collected by the terminal, and use the image frame of the current moment and the visual image frame of the next moment after the current moment and adjacent to the current moment to perform mutual verification until the verification result is a verification success.
[0130] Step S211: When the verification result is successful, the server generates the transformation matrix using the first pose information and the first virtual pose information.
[0131] In some embodiments, see Figure 6 , Figure 6 It is shown that generating the conversion matrix in step S211 can be achieved by following steps S2111 to S2116:
[0132] Step S2111, obtaining the first reference information of the geographic coordinate system.
[0133] Here, the first reference information includes a reference orientation and a reference position of the geographic coordinate system.
[0134] Step S2112: Perform plane projection processing on the geographic coordinate system based on the first reference information to obtain the geographic coordinate system after plane projection processing.
[0135] The geographic coordinate system defines the geographic location information and orientation information corresponding to the visual image frame through a spherical surface. The range corresponding to the geographic coordinate system is relatively large. Therefore, the geographic coordinate system can be plane-projected based on the first reference information to obtain the geographic coordinate system after plane projection processing, so as to reduce the range corresponding to the geographic coordinate system.
[0136] Step S2113, obtaining the second reference information of the geographic coordinate system after plane projection processing.
[0137] Here, the second reference information includes a reference orientation and a reference position of the geographic coordinate system after plane projection processing.
[0138] Step S2114: converting the geographic coordinate system after the plane projection processing into a Cartesian coordinate system by adjusting the second reference information.
[0139] Since the VIO coordinate system defines the geographic location information and orientation information corresponding to the visual image frame through a rectangular coordinate system, which is different from the definition method of the world coordinate system, in order to establish the relationship between the geographic coordinate system after plane projection processing and the VIO coordinate system, the server can convert the geographic coordinate system after plane projection processing into a Cartesian coordinate system. The Cartesian coordinate system can define the geographic location information and orientation information corresponding to the visual image frame through a rectangular coordinate system, and the reference information of the Cartesian coordinate system is the same.
[0140] Here, after determining the reference information of the Cartesian coordinate system, the server may adjust the second reference information so that the second reference information is consistent with the reference information of the Cartesian coordinate system, so as to transform the geographic coordinate system after the plane projection processing into the Cartesian coordinate system.
[0141] Step S2115, obtaining the third reference information of the Cartesian coordinate system.
[0142] Step S2116: based on the third reference information, convert the first posture information into the second posture information.
[0143] Here, based on the third reference information of the Cartesian coordinate system, the server can convert the posture information of the visual image frame at the current moment in the visual coordinate system relative to the first reference information (i.e., the first posture information) into the posture information of the visual image frame at the current moment in the Cartesian coordinate system relative to the third reference information (i.e., the second posture information).
[0144] Step S2116, using the second posture information and the first virtual posture to obtain a transformation matrix.
[0145] Since the Cartesian coordinate system and the VIO coordinate system have the same definition, for the same visual image frame at the current moment, the server can obtain the transformation matrix T by determining the rotation relationship between the second pose information and the first virtual pose.
[0146] In step S212, the server uses a transformation matrix to transform the second virtual posture information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, to obtain the transformed second virtual posture information.
[0147] Here, the second virtual posture information after the conversion process includes the position information and orientation information in the geographic coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment.
[0148] For the next moment after the current moment and adjacent to the current moment, the terminal can collect the visual image frame of the next moment after the current moment and adjacent to the current moment through the camera; the visual image frame of the next moment after the current moment and adjacent to the current moment can be automatically collected by the positioning application according to preset conditions, or collected by the user by clicking a button on the client of the positioning application. Then, the terminal can render the visual image frame of the next moment after the current moment and adjacent to the current moment, so as to display the rendered visual image frame of the next moment after the current moment and adjacent to the current moment on the current interface of the positioning application for the user to view.
[0149] In addition, the terminal can also send the collected visual image frame of the next moment after the current moment and adjacent to the current moment to the server, so that the server can obtain the second virtual posture information relative to the reference posture in the VIO coordinate system corresponding to the visual image frame of the next moment after the current moment and adjacent to the current moment based on the VIO function. Then, the server uses the transformation matrix to transform the second virtual posture information to determine the real posture information of the terminal at the next moment, without having to rely on the auxiliary position and three-dimensional point cloud map for positioning at the next moment, avoiding the influence of transmission delay and sensor accuracy on positioning accuracy, and improving positioning accuracy.
[0150] In some embodiments, in an application scenario where path navigation is required, the user can also input the target location information of this path navigation through the positioning application on the terminal, and the input method includes but is not limited to text input and voice input; after the server receives the target location information of this path navigation sent by the terminal, it determines the path information based on the second virtual posture information after the conversion and the target location information; the path information includes guidance information for reaching the target location corresponding to the target location information.
[0151] Here, the target location information can be the target location in a specific environment that the user needs to reach when in a specific environment. For example, the target location information can be the target location in an indoor building that the user needs to reach when in an indoor building, or it can be the target location in a scenic area that the user needs to reach when in a scenic area.
[0152] In an embodiment of the present application, during the process of determining the path information, the server obtains the model information corresponding to the visual image frame at the current moment; based on the model information corresponding to the visual image frame at the current moment, a virtual scene in the VIO coordinate system is generated; based on the first pose information and the target position information, guidance information for reaching the target position from the position corresponding to the first pose information is determined; through a transformation matrix, the guidance information for reaching the target position from the position corresponding to the first pose information is fused into the virtual scene to obtain a fused virtual scene; based on the second virtual pose information after the conversion processing, guidance information for reaching the target position corresponding to the target position information is determined from the fused virtual scene.
[0153] Here, the model information corresponding to the visual image frame at the current moment includes POI, objects, scenes and other information in the geographic coordinate system corresponding to the visual image frame at the current moment; the virtual scene can be a three-dimensional virtual scene, which can provide users with POIs, guide signs or guide paths with differences in distance and size, thereby enhancing the user's immersion and sense of reality in the path navigation and positioning process.
[0154] The server can initialize and generate a virtual scene in the VIO coordinate system based on the reference information of the VIO coordinate system; then, use the transformation matrix to transform the model information corresponding to the visual image frame at the current moment to obtain the model information in the VIO coordinate system corresponding to the visual image frame at the current moment, and fuse the model information in the VIO coordinate system corresponding to the visual image frame at the current moment into the virtual scene; then, the server can fuse the guidance information used to reach the target position from the position corresponding to the first posture information into the virtual scene through the transformation matrix to obtain the fused virtual scene; finally, for the next moment after the current moment and adjacent to the current moment, determine the guidance information used to reach the target position corresponding to the target position information from the fused virtual scene.
[0155] In the above process, as time changes, the model information corresponding to the visual image frame may change, that is, the user's visual pose information in the world coordinate system may change; the visual pose information may include visual position and visual orientation, such as camera position and rotation angle; in order to ensure the accuracy of the superposition of guidance information and real visual image frames, it is necessary to ensure that the visual pose information in the virtual scene is consistent with the user's visual pose information in the world coordinate system.
[0156] In some embodiments, the terminal may obtain model information corresponding to a visual image frame at a next moment after the current moment and adjacent to the current moment; when the model information corresponding to the visual image frame at the current moment is different from the model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, it is determined that the visual pose information of the visual image frame at the next moment after the current moment and adjacent to the current moment in the world coordinate system has changed; then, based on the model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, the visual pose information corresponding to the fused virtual scene is updated to obtain an updated virtual scene; based on the second virtual pose information after the conversion processing, the guidance information for reaching the target position corresponding to the target position information is determined from the updated virtual scene. That is, by detecting the changes in the model information corresponding to the visual image frames at different moments, the visual pose information in the virtual scene is updated in real time, so that the pose information of the guidance information is consistent with the pose information of the real visual image frame.
[0157] By acquiring the visual pose information of the visual image frame and updating the visual pose information of the virtual scene in real time, the accuracy of the superposition of the virtual guidance information and the real visual image frame can be guaranteed, thereby improving the navigation effect.
[0158] Step S213: The server sends the converted second virtual posture information to the terminal.
[0159] In some embodiments, the server may also send guidance information for reaching the target location corresponding to the target location information to the terminal.
[0160] Step S214: the terminal displays the converted second virtual posture information on the current interface.
[0161] In some embodiments, the terminal may also display guidance information for reaching the target location corresponding to the target location information on the current interface.
[0162] The positioning method provided in the embodiment of the present application, in response to the positioning request, obtains the first pose information of the terminal, and then obtains the first virtual pose information in the VIO coordinate system corresponding to the visual image frame at the current moment; then, the first pose information and the first virtual pose information are used to generate a conversion matrix; finally, the second virtual pose information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment is converted by the conversion matrix to obtain the second virtual pose information after the conversion. In the positioning process, the conversion matrix is first obtained by the real positioning information of the server at the current moment and the VIO virtual positioning information, and then the VIO virtual pose information (including position information and orientation) at the next moment is converted by the conversion matrix to obtain the real pose information at the next moment, without relying on the server-side positioning engine and the terminal sensor to locate the next moment, avoiding the influence of transmission delay and sensor accuracy on the positioning accuracy, and improving the positioning accuracy. In addition, since the conversion matrix at the next moment is determined based on the current moment, the server positioning can be performed intermittently during the positioning process to update the conversion matrix, further improving the positioning accuracy.
[0163] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.
[0164] In the current AR navigation solution under the mini program, in terms of the way to achieve the combination of virtual and real, POIs and logos are only used as icons, and are proportionally superimposed on the camera image according to the relative positions of front, back, left and right, and the rendering of three-dimensional scenes and real camera images is not realized. In addition, during the positioning process, the comprehensive positioning engine on the server side can return the position information, and the direction angle information is obtained by relying on the mobile phone compass, which will have the following two problems:
[0165] On the one hand, there may be electromagnetic interference in the indoor environment, which will affect the direction of the mobile phone compass, resulting in inaccurate angular information. At the same time, the frame rate returned by the compass in the mini program is insufficient, which will also cause inaccurate angular information when the mobile phone moves quickly. On the other hand, in the process of positioning through the integrated positioning engine on the server side, the impact of the round-trip transmission time when sending a positioning request to the server is not considered, resulting in the possibility of untimely positioning in areas with poor network; in addition, when sending the acquired camera image to the server for image recognition, it is not considered that there may be network delays during the image transmission process and that the current mobile phone orientation and displacement may have changed. Taking all the above factors into consideration, there will be a situation where the virtual information does not match the actual position and orientation of the camera in the real world, resulting in low AR navigation accuracy and poor user immersion experience.
[0166] The present application embodiment provides a positioning method, which is applied to indoor AR walking navigation scenarios, such as Figure 7 As shown, when performing indoor AR navigation under the applet, in the visual positioning process 71, the current camera image is obtained through the applet camera API, and transmitted to the visual positioning server for the first visual positioning, the geographic location is determined and the camera reference point is established. Then, in the relative position positioning process 72, the applet-side VIO is used for relative position positioning to establish a geographic reference point conversion relationship matrix. Subsequently, the geographic reference point conversion relationship matrix is used to convert the current camera pose matrix returned by the applet-side VIO into geographic coordinates for positioning, and the user is instructed to reach the destination through virtual information. In addition, in the similar area identification process 73, by introducing Bluetooth and WIFI positioning services, the problems of cross-layer and similar area positioning errors of pure visual positioning solutions can also be improved. In this way, the positioning reference point error caused by sensor accuracy and transmission delay problems in the traditional applet environment can be effectively improved, resulting in the mismatch between the real world and virtual information, and the poor AR realism and immersion experience. The problems, that is, the positioning accuracy is improved, the superposition accuracy of virtual information and the real image of the camera is improved, and the intermediate positioning is more real-time, so that the user has a better immersive experience. In addition, by introducing Bluetooth and WIFI positioning services, the problems of cross-layer and similar area positioning errors in pure visual positioning solutions can be improved, further improving the positioning accuracy.
[0167] The implementation process of the positioning method provided in the embodiment of the present application includes the following steps S301 to S307:
[0168] Step S301: After the mini program uses a robot to collect images of the building that needs AR navigation, it uses SFM technology to process the collected images into camera visual image frames, establishes a three-dimensional image point cloud map, and builds a visual positioning server based on image retrieval and feature matching visual positioning technology.
[0169] In step S302, the mini program collects WIFI fingerprints or Bluetooth information in the building so that the floor and area can be judged later through the WIFI fingerprints or Bluetooth information collected on the mini program.
[0170] Step S303, using the VIO capability of the visual API on the mini program side, initialize a three-dimensional rendering scene consistent with the VIO coordinate system, and render the visual image frame into the three-dimensional rendering scene.
[0171] Step S304: During the first positioning, the mini program end sends the visual image frame at the current moment to the visual positioning server for visual positioning; after the visual positioning server performs image retrieval and feature matching based on the visual image frame at the current moment, it returns the pose information (including geographic location and orientation information) of the visual image frame at the current moment to the mini program end, and the mini program end adopts a two-request verification mechanism to eliminate the situation where the deflection error is large, that is, the situation where the orientation of the virtual visual return point has a large error with the actual camera orientation.
[0172] Figure 8 is a schematic diagram of the process of the first visual positioning provided by the embodiment of the present application, such as Figure 8 As shown, first, the mini program side obtains the visual image frame at the current moment, and performs Canvas rendering processing 801 on the visual image frame at the current moment, obtains the visual image frame at the current moment after Canvas rendering processing, and displays it on the current interface of the terminal; then, the mini program side converts the visual image frame at the current moment after Canvas rendering processing into an image in JPG format 802, encapsulates it into a positioning request, and sends it to the visual positioning server; after receiving the positioning request, the visual positioning server returns the pose information in the GCJ02 coordinate system corresponding to the visual image frame at the current moment to the mini program side based on image retrieval and feature matching, so that the mini program side can subsequently calculate the transformation matrix 803 according to the pose information in the GCJ02 coordinate system corresponding to the visual image frame at the current moment; wherein, the pose information in the GCJ02 coordinate system corresponding to the visual image frame at the current moment includes the GCJ02 coordinates compared to the north heading angle (i.e., the reference heading information); the transformation matrix 803 is used to realize the conversion between the VIO coordinate system and the GCJ02 coordinate system.
[0173] During the first visual positioning process, there may be a situation where the orientation returned by the visual positioning server is large in error with the actual orientation of the camera, resulting in a positioning error. In response to this, the embodiment of the present application adds a secondary verification process, and two requests are used for mutual verification during the first positioning, thereby improving the accuracy of the first positioning. First, for the first visual image frame corresponding to the current moment, after the applet end receives the response (Response) of the successful positioning of the first visual image frame, it immediately or at a short interval sends a positioning request to the visual positioning server again to verify the second visual image frame corresponding to the next moment; then, after the applet end receives the response of the successful positioning of the second visual image frame, the posture information returned by the second visual image frame is checked to be consistent with the posture information returned by the first visual image frame and the position information (such as longitude and latitude) and orientation of the first visual image frame and the second visual image frame obtained based on the VIO function; if not consistent, it means that there is a positioning error or the VIO function has an unstable problem, and the third visual image frame and the second visual image frame can be verified in the above manner, or the first visual positioning process can be re-executed; if consistent, it means that the verification is successful, and step S305 can be executed. If the mini program side does not receive a successful response to the second visual image frame positioning, the mini program side can send a positioning request to the visual positioning server again until it receives a successful positioning response returned by the visual positioning server and the mutual verification with the first visual image frame is successful. If the verification is successful, the mini program side will carry the context information of the pose information returned by the second visual image frame when sending a positioning request to the visual positioning server next time, so that the visual positioning server can determine that it is continuous positioning.
[0174] In addition, when using the visual positioning server for positioning, there may be errors in the recognition of similar scenes such as cross-layers or elevator entrances, resulting in inaccurate positioning. In this regard, the embodiment of the present application adds a positioning process for similar scenes such as cross-layers and elevator entrances. After the mini program collects the Bluetooth and WIFI information of the building that needs AR navigation, the collected information is sent to the visual positioning server as an auxiliary parameter, so that the visual positioning server performs positioning based on the auxiliary parameters and returns the posture information. In this way, the problem of incorrect recognition in similar scenes can be effectively improved, and the positioning accuracy can be improved.
[0175] Step S305: Based on the VIO capability of the visual API of the mini program, the mini program locates the virtual posture of the visual image frame at the current moment, and establishes a relative position geographic reference point conversion relationship matrix through the first positioning, so as to subsequently use the conversion relationship matrix to convert the posture information corresponding to the visual image frame at subsequent moments into posture information under geographic coordinates for positioning and navigation; wherein, VIO can return a relatively accurate relative position within a certain interval.
[0176] Fig. 9 is a schematic diagram of a VIO coordinate system and a geographic coordinate system provided in an embodiment of the present application, such as Fig. 9 As shown in the figure, the geographic coordinate system defines the real pose information corresponding to the visual image frame at the current moment through a spherical surface, and the VIO coordinate system defines the information corresponding to the visual image frame at the current moment through a rectangular coordinate system. Therefore, it is necessary to calculate the transformation matrix to realize the conversion between the geographic coordinate system and the VIO coordinate system.
[0177] Fig.10 is a schematic diagram of a flow chart of calculating a conversion matrix provided in an embodiment of the present application, such as Fig.10 As shown, after the mini program determines the reference information 1001 of the GCJ02 coordinate system, the GCJ02 coordinate system can be converted into the WG584 geographic coordinate system 1002 through a correction algorithm; the WG584 geographic coordinate system is subjected to plane projection processing to obtain the WG584 geographic coordinate system 1003 after plane projection processing; based on the reference information of the WG584 geographic coordinate system after plane projection processing, the pose information under the local Cartesian coordinate system 1004 corresponding to the visual image frame at the current moment is obtained; combined with the pose information under the VIO coordinate system 1005 corresponding to the visual image frame at the current moment, the transformation matrix can be calculated.
[0178] In addition, after a certain time interval (such as 5 seconds), the mini program side can resend a positioning request to the server side. At the same time, the mini program side can locate the relative position of the VIO every 1 second to update the transformation matrix.
[0179] Step S306: The mini program obtains a coordinate point string corresponding to the target location based on the route planning service, and draws the coordinate point string into the three-dimensional rendering scene as guidance information.
[0180] In the embodiment of the present application, in order to make the user more immersed, a rendering process of a three-dimensional scene and a real camera image can be added. Based on the route planning service, the mini program end can obtain the coordinate point string in the geographic coordinate system corresponding to the target location by combining the visual image frame at the current moment and the target location of this path navigation; then, through the conversion matrix, the coordinate point string is drawn into the three-dimensional rendering scene as guidance information; the guidance information includes route guidance, identification, PIO and other virtual guidance information.
[0181] Fig.11 is a flow chart of a scene rendering process 1100 provided in an embodiment of the present application, such as Fig.11As shown, after the mini program determines the reference information of the GCJ02 coordinate system, the GCJ02 coordinate system is converted into the WG584 geographic coordinate system through a correction algorithm; the WG584 geographic coordinate system is subjected to plane projection processing to obtain the WG584 geographic coordinate system after plane projection processing; based on the reference information of the WG584 geographic coordinate system after plane projection processing, the WG584 geographic coordinate system after plane projection processing is converted into a Cartesian coordinate system; using the transformation matrix, the coordinate point string under the Cartesian coordinate system is drawn as guidance information into the three-dimensional rendering scene under the VIO coordinate system.
[0182] Step S307, the mini program detects the change of the VIO view matrix and updates the visual pose information (i.e., camera coordinates and orientation angle) in the 3D rendering scene in real time.
[0183] In an embodiment of the present application, the model information contained in the visual image frames at different moments may be different. The mini program end can detect the model information contained in the visual image frame at the current moment and the visual image frame at the next moment adjacent to the current moment after the current moment, so as to determine whether the view matrix of the VIO has changed. In the case of whether the view matrix of the VIO has changed, the camera coordinates and orientation in the three-dimensional rendering scene are updated according to the model information contained in the visual image frame at the next moment adjacent to the current moment after the current moment, so that the guidance information can match the visual image frame at the next moment adjacent to the current moment after the current moment, thereby ensuring the accuracy of the superposition of the guidance information and the visual image frame at the next moment adjacent to the current moment after the current moment.
[0184] Step S308, looping through steps S304 to S307 until the target location of this path navigation is reached.
[0185] In an embodiment of the present application, when the user moves towards the target position based on the guidance information, the mini program end can detect the second virtual posture information of the visual image frame at the next moment adjacent to the current moment after the current moment in the VIO coordinate system; transform the second virtual posture information based on the transformation matrix to obtain the transformed second virtual posture information; based on the transformed second virtual posture information and the target position information, it can determine whether the user has reached the target position.
[0186] Fig.12 is a flowchart of a navigation process 1200 provided in an embodiment of the present application, such as Fig.13As shown, the mini program uses the inverse matrix of the transformation matrix to transform the VIO coordinate system to obtain the Cartesian coordinate system; then, by adjusting the reference information of the local Cartesian coordinate system, the Cartesian coordinates are transformed into the WG584 geographic coordinates of the plane projection; then, through the reverse projection algorithm, the WG584 geographic coordinates of the plane projection are reversely projected to obtain the WG584 geographic coordinate system; finally, through the deflection algorithm, the GCJ02 coordinate system is transformed into the WG584 geographic coordinate system to obtain the second virtual pose information after the transformation in the WG584 geographic coordinate system.
[0187] Fig.13 is a path diagram of the positioning method provided in the embodiment of the present application, such as Fig.13 As shown, during navigation from the user's current location (F1) to the target location (F3), guidance signs, guidance paths and the like may be displayed in the visual image frame at the current moment to generate path information for the user for this path navigation.
[0188] Fig.14 is a flow chart of a positioning method with scene rendering provided in an embodiment of the present application, such as Fig.14 As shown, first, according to the first positioning process 141, the processed visual image frame is sent to the server to request the pose information of the visual image frame at the current moment. Then, after the pose information is verified, a transformation matrix is established according to the verified pose information and the pose matrix in the VIO coordinate system. In this way, the situation where the pose information has a large deflection error can be eliminated and the accuracy of the first positioning can be improved. In the positioning process 142 on the mini program side, the mini program side determines the virtual geographic coordinates and orientation of the visual image frame at the subsequent moment in the VIO coordinate, and transforms the virtual geographic coordinates and orientation of the visual image frame at the subsequent moment in the VIO coordinate through the transformation matrix to perform positioning and generate path information to the target location for the user.
[0189] In an embodiment of the present application, in order to increase the realism and immersion of the user's AR navigation, a rendering initialization process 143 and a three-dimensional scene update process 144 are added in the process of generating path information. After the mini program generates a three-dimensional scene based on the visual image frame, the visual posture information in the three-dimensional scene is updated in real time based on the view matrix corresponding to the visual image frame at subsequent moments, so that the user can perceive the differences in distance or size of the indication signs and indication paths in different locations, thereby improving the immersion and realism of the user's AR navigation.
[0190] It can be understood that in the embodiments of the present application, the content involving user information, such as visual image frames and target location information collected by the terminal, if it involves data related to user information or corporate information, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0191] The following is a description of an exemplary structure of the positioning device 354 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2 As shown, the positioning device 354 includes: a first acquisition module 3541, used to obtain the first posture information of the terminal in response to a positioning request; the positioning request includes the visual image frame at the current moment collected by the terminal; the first posture information is the real posture information in the geographic coordinate system corresponding to the visual image frame at the current moment; a second acquisition module 3543, used to obtain the first virtual posture information in the VIO coordinate system corresponding to the visual image frame at the current moment; a generation module 3544, used to generate a transformation matrix using the first posture information and the first virtual posture information; a processing module 3545, used to use the transformation matrix to transform the second virtual posture information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, to obtain the second virtual posture information after the transformation; the second virtual posture information after the transformation is used to determine the real posture information of the terminal at the next moment.
[0192] In some embodiments, the device also includes: a conversion module, used to obtain first reference information of a geographic coordinate system; the first reference information includes reference orientation information and reference position information of the geographic coordinate system; based on the first reference information, the first posture information is converted into second posture information; the second posture information is the real posture information in the Cartesian coordinate system corresponding to the visual image frame at the current moment; the processing module is also used to generate the transformation matrix using the second posture information and the first virtual posture.
[0193] In some embodiments, the conversion module is also used to: perform plane projection processing on the geographic coordinate system based on the first reference information to obtain the geographic coordinate system after the plane projection processing; obtain second reference information of the geographic coordinate system after the plane projection processing; the second reference information includes the reference orientation information and reference position information of the geographic coordinate system after the plane projection processing; by adjusting the second reference information, the geographic coordinate system after the plane projection processing is converted into a Cartesian coordinate system; obtain third reference information of the Cartesian coordinate system; the third reference information includes the reference orientation information and reference position information of the Cartesian coordinate system; based on the third reference information, convert the first posture information into second posture information.
[0194] In some embodiments, the device also includes: a verification module, used to obtain third posture information of the terminal; the third posture information is the real posture information under the geographic coordinates corresponding to the visual image frame of the previous moment before the current moment and adjacent to the current moment; the third posture information and the first posture information are verified to obtain a verification result; the generation module is also used to: when the verification result is a successful verification, use the first posture information and the first virtual posture information to generate the transformation matrix.
[0195] In some embodiments, the verification module is also used to: determine the real posture deviation using the first posture information and the third posture information; obtain the third virtual posture information in the VIO coordinate system corresponding to the visual image frame of the previous moment before the current moment and adjacent to the current moment; determine the virtual posture deviation using the first virtual posture information and the third virtual posture information; and when the real posture deviation matches the virtual posture deviation, determine that the verification result is a successful verification.
[0196] In some embodiments, the device also includes: a path generation module, which is used to receive the target position information of this positioning sent by the terminal; determine the path information based on the second virtual posture information after the conversion processing and the target position information; the path information includes guidance information for reaching the target position corresponding to the target position information.
[0197] In some embodiments, the path generation module is also used to: obtain model information corresponding to the visual image frame at the current moment; generate a virtual scene in the VIO coordinate system based on the model information corresponding to the visual image frame at the current moment; determine guidance information for reaching the target position from the position corresponding to the first posture information based on the first posture information and the target position information; fuse the guidance information for reaching the target position from the position corresponding to the first posture information into the virtual scene through the transformation matrix to obtain a fused virtual scene; and determine guidance information for reaching the target position corresponding to the target position information from the fused virtual scene based on the second virtual posture information after the conversion processing.
[0198] In some embodiments, the path generation module is also used to: obtain model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment; when the model information corresponding to the visual image frame at the current moment is different from the model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, based on the model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, update the visual posture information corresponding to the fused virtual scene to obtain an updated virtual scene; based on the second virtual posture information after the conversion processing, determine the guidance information for reaching the target position corresponding to the target position information from the updated virtual scene.
[0199] In some embodiments, the positioning request also includes auxiliary feature information of the real position corresponding to the visual image frame at the current moment; the auxiliary feature information is used to distinguish similar positions; the first acquisition module is also used to: respond to the positioning request, determine the auxiliary position information corresponding to the visual image frame at the current moment based on the auxiliary feature information; perform feature matching on the visual image frame at the current moment with a preset three-dimensional point cloud map to obtain a matching result; determine the first posture information based on the auxiliary position information and the matching result.
[0200] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.
[0201] The embodiment of the present application provides a computer program product, which includes an executable instruction, which is a computer instruction; the executable instruction is stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instruction from the computer-readable storage medium and the processor executes the executable instruction, the electronic device executes the method described in the embodiment of the present application.
[0202] The present application embodiment provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the present application embodiment, for example, Figure 3 The method shown.
[0203] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPR OM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above memories.
[0204] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0205] As an example, executable instructions may, but need not correspond to, a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.
[0206] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. A positioning method, It is characterized in that The method comprises: In response to a positioning request, obtaining first posture information of the terminal; the positioning request includes a visual image frame at a current moment collected by the terminal; the first posture information is real posture information in a geographic coordinate system corresponding to the visual image frame at the current moment; Obtaining first virtual pose information in a visual inertial odometer (VIO) coordinate system corresponding to the visual image frame at the current moment; Generate a transformation matrix using the first posture information and the first virtual posture information; The transformation matrix is used to transform the second virtual posture information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment to obtain the transformed second virtual posture information; the transformed second virtual posture information is used to determine the real posture information of the terminal at the next moment.
2. The method according to claim 1, It is characterized in that The step of generating a transformation matrix by using the first pose information and the first virtual pose information includes: Acquire first reference information of a geographic coordinate system; the first reference information includes reference orientation information and reference position information of the geographic coordinate system; Based on the first reference information, the first posture information is converted into second posture information; the second posture information is the real posture information in the Cartesian coordinate system corresponding to the visual image frame at the current moment; The transformation matrix is generated using the second posture information and the first virtual posture.
3. The method according to claim 2, It is characterized in that The converting the first posture information into second posture information based on the first reference information includes: Performing plane projection processing on the geographic coordinate system based on the first reference information to obtain a geographic coordinate system after plane projection processing; Acquire second reference information of the geographic coordinate system after the plane projection processing; the second reference information includes reference orientation information and reference position information of the geographic coordinate system after the plane projection processing; By adjusting the second reference information, the geographic coordinate system after the plane projection processing is converted into a Cartesian coordinate system; Acquire third reference information of the Cartesian coordinate system; the third reference information includes reference orientation information and reference position information of the Cartesian coordinate system; Based on the third reference information, the first posture information is converted into second posture information.
4. The method according to claim 1, It is characterized in that The step of generating a transformation matrix by using the first posture information and the first virtual posture information includes: Acquire third posture information of the terminal; the third posture information is real posture information under geographical coordinates corresponding to a visual image frame at a previous moment before the current moment and adjacent to the current moment; Performing verification processing on the third posture information and the first posture information to obtain a verification result; When the verification result is successful, the transformation matrix is generated using the first posture information and the first virtual posture information.
5. The method according to claim 4, It is characterized in that The verifying the third posture information and the first posture information to obtain a verification result includes: Determine a true posture deviation using the first posture information and the third posture information; Acquire third virtual pose information in the VIO coordinate system corresponding to the visual image frame of the previous moment before the current moment and adjacent to the current moment; Determine a virtual posture deviation using the first virtual posture information and the third virtual posture information; In a case where the real posture deviation matches the virtual posture deviation, the verification result is determined to be a successful verification.
6. The method according to claim 1, It is characterized in that The method further comprises: Receiving target location information of this route navigation sent by the terminal; Based on the converted second virtual posture information and the target position information, path information is determined; the path information includes guidance information for reaching the target position corresponding to the target position information.
7. The method according to claim 6, It is characterized in that The determining of path information based on the converted second virtual posture information and the target position information includes: Obtaining model information corresponding to the visual image frame at the current moment; Generate a virtual scene in a VIO coordinate system based on the model information corresponding to the visual image frame at the current moment; Based on the first posture information and the target position information, determining guidance information for reaching the target position from the position corresponding to the first posture information; By means of the transformation matrix, the guidance information for reaching the target position from the position corresponding to the first posture information is integrated into the virtual scene to obtain a integrated virtual scene; Based on the converted second virtual posture information, guidance information for reaching the target position corresponding to the target position information is determined from the fused virtual scene.
8. The method according to claim 7, It is characterized in that The step of determining, based on the converted second virtual posture information, from the fused virtual scene, guidance information for reaching the target position corresponding to the target position information comprises: Acquire model information corresponding to a visual image frame at a next moment after the current moment and adjacent to the current moment; When the model information corresponding to the visual image frame at the current moment is different from the model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, the visual pose information corresponding to the fused virtual scene is updated based on the model information corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment to obtain an updated virtual scene; Based on the converted second virtual posture information, guidance information for reaching the target position corresponding to the target position information is determined from the updated virtual scene.
9. The method according to any one of claims 1 to 8, It is characterized in that The positioning request also includes auxiliary feature information of the real position corresponding to the visual image frame at the current moment; the auxiliary feature information is used to distinguish similar positions; The step of obtaining the first position information of the terminal in response to the positioning request includes: In response to the positioning request, determining auxiliary position information corresponding to the visual image frame at the current moment based on the auxiliary feature information; Perform feature matching between the visual image frame at the current moment and the preset three-dimensional point cloud map to obtain a matching result; Based on the auxiliary position information and the matching result, the first position information is determined.
10. A positioning device, It is characterized in that The device comprises: A first acquisition module, configured to acquire first posture information of the terminal in response to a positioning request; the positioning request includes a visual image frame at a current moment collected by the terminal; the first posture information is real posture information in a geographic coordinate system corresponding to the visual image frame at the current moment; A second acquisition module is used to acquire first virtual pose information in a VIO coordinate system corresponding to the visual image frame at the current moment; A generating module, configured to generate a transformation matrix using the first posture information and the first virtual posture information; A processing module is used to use the transformation matrix to transform the second virtual posture information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, so as to obtain the second virtual posture information after the transformation; the second virtual posture information after the transformation is used to determine the real posture information of the terminal at the next moment.
11. An electronic device, It is characterized in that include: A memory for storing executable instructions; A processor, used to implement the positioning method described in any one of claims 1 to 9 when executing the executable instructions stored in the memory.
12. A computer-readable storage medium, It is characterized in that Executable instructions are stored, which are used to cause a processor to execute the executable instructions to implement the positioning method described in any one of claims 1 to 9.
Citation Information
Cited By
Positioning method and apparatus, electronic device, computer-readable storage medium, and computer program product
EP4622363A1
