Positioning method and apparatus, electronic device, computer-readable storage medium, and computer program product
By generating and using the transformation matrix, the impact of transmission delay and sensor accuracy on positioning is solved, and the positioning accuracy and user experience of augmented reality indoor navigation is improved.
Patent Information
- Application Number
- PCT/CN2024/085522
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-04-02
- Publication Date
- 2024-12-05
AI Technical Summary
In augmented reality indoor navigation, the prior art fails to effectively consider the impact of transmission delay and sensor accuracy on positioning accuracy, resulting in large positioning errors and affecting navigation effects.
By acquiring the visual image frame of the terminal, a conversion matrix is generated to convert the virtual position information under the visual inertial odometer coordinate system, which is directly used to determine the real position of the terminal, reducing dependence on the server positioning engine and sensor, and reducing transmission time Delay and precision influence.
It improves the accuracy of positioning, reduces positioning errors, and enhances the real-timeness of navigation and user immersion.
Smart Images

Figure CN2024085522_05122024_PF_FP_ABST
Abstract
Description
Positioning method, device, electronic device, computer-readable storage medium, and computer program product
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on the Chinese patent application with application number 202310638131.1 and application date of May 31, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field
[0003] The embodiments of the present application relate to the field of the Internet, and relate to but are not limited to a positioning method, device, electronic device, computer-readable storage medium, and computer program product. Background Art
[0004] Currently, in the field of augmented reality (AR) indoor navigation, the positioning process usually relies on a server-side integrated positioning engine to determine the geographic location, and the directional angle information when determining the geographic location is achieved by relying on mobile phone sensors (such as a compass).
[0005] However, the server-side integrated positioning engine doesn't account for the impact of transmission latency on positioning accuracy when performing positioning. Furthermore, when obtaining angular information from mobile phone sensors, the impact of low sensor accuracy on angular information is also not considered. This leads to large positioning errors during indoor navigation, resulting in poor navigation results.
[0006] Summary of the Invention
[0007] The embodiments of the present application provide a positioning method, device, electronic device, computer-readable storage medium, and computer program product, which can be applied at least to the fields of maps and map positioning, and can effectively improve the problem of large positioning errors caused by low positioning sensor accuracy and positioning delay, thereby improving the accuracy of positioning.
[0008] The technical solution of the embodiment of the present application is implemented as follows:
[0009] An embodiment of the present application provides a positioning method, which is executed by an electronic device and includes: obtaining first posture information of a terminal in response to a positioning request; the positioning request includes a visual image frame at a current moment collected by the terminal; the first posture information is the real posture information of the visual image frame at the current moment in a geographic coordinate system; obtaining first virtual posture information of the visual image frame at the current moment in a visual inertial odometry (VIO) coordinate system; generating a transformation matrix based on the first posture information and the first virtual posture information; using the transformation matrix to transform the second virtual posture information of the visual image frame at a next moment after the current moment and adjacent to the current moment in the VIO coordinate system to obtain the transformed second virtual posture information; and determining the real posture information of the terminal at the next moment based on the transformed second virtual posture information.
[0010] An embodiment of the present application provides a positioning device, which includes: a first acquisition module, configured to acquire first posture information of a terminal in response to a positioning request; the positioning request includes a visual image frame at a current moment collected by the terminal; the first posture information is the real posture information of the visual image frame at the current moment in a geographic coordinate system; a second acquisition module, configured to acquire first virtual posture information of the visual image frame at the current moment in a VIO coordinate system; a generation module, configured to generate a transformation matrix based on the first posture information and the first virtual posture information; a processing module, configured to use the transformation matrix to transform the second virtual posture information of the visual image frame at a next moment after the current moment and adjacent to the current moment in the VIO coordinate system to obtain the transformed second virtual posture information; a determination module, configured to determine the real posture information of the terminal at the next moment based on the transformed second virtual posture information.
[0011] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned positioning method when executing the executable instructions stored in the memory.
[0012] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein a processor of an electronic device reads the executable instructions from the computer-readable storage medium and implements the above-mentioned positioning method when executing the executable instructions.
[0013] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned positioning method.
[0014] The embodiments of the present application have the following beneficial effects: upon receiving a positioning request including a visual image frame at the current moment, in response to the positioning request, the first position information of the terminal is obtained, as well as the first virtual position information of the visual image frame at the current moment in the VIO coordinate system, and a transformation matrix is generated using the first position information and the first virtual position information; the second virtual position information of the visual image frame at the next moment after the current moment and adjacent to the current moment in the VIO coordinate system is transformed using the transformation matrix to obtain the second virtual position information after the transformation. After obtaining the transformation matrix through the real positioning information of the server at the current moment and the VIO virtual positioning information, during the positioning process, the transformation matrix is directly used to transform the VIO virtual position information at the next moment, thereby obtaining the real position information of the terminal at the next moment, without relying on the server-side positioning engine and the terminal sensor to perform positioning at the next moment. In this way, the transmission delay can be greatly reduced, and the influence of sensor accuracy on positioning accuracy can be reduced, thereby improving positioning accuracy. In addition, since the conversion matrix at the next moment is determined based on the current moment, server-side positioning can be performed intermittently during the positioning process, thereby achieving continuous updating of the conversion matrix and further improving the positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] FIG1 is a schematic diagram of an optional architecture of a positioning system provided in an embodiment of the present application;
[0016] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;
[0017] FIG3 is a schematic diagram of an optional flow chart of a positioning method provided in an embodiment of the present application;
[0018] FIG4 is another optional flowchart of the positioning method provided in an embodiment of the present application;
[0019] FIG5 is a schematic diagram of an implementation flow of determining the first pose information provided in an embodiment of the present application;
[0020] FIG6 is a schematic diagram of an implementation flow of generating a conversion matrix according to an embodiment of the present application;
[0021] FIG7A is a schematic diagram of an implementation flow of a positioning method in an AR walking navigation scenario provided by an embodiment of the present application;
[0022] FIG7B is a schematic diagram of an implementation flow of a positioning method for a mini-program provided in an embodiment of the present application;
[0023] FIG8 is a schematic diagram of an implementation flow of generating a positioning request according to an embodiment of the present application;
[0024] FIG9 is a schematic diagram of the architecture of the world coordinate system and the VIO coordinate system provided in an embodiment of the present application;
[0025] FIG10 is a schematic diagram of another implementation flow of generating a conversion matrix provided in an embodiment of the present application;
[0026] FIG11 is a schematic diagram of a scene rendering process according to an embodiment of the present application;
[0027] FIG12 is a schematic diagram of a positioning process according to an embodiment of the present application;
[0028] FIG13 is a schematic diagram of an application scenario of the positioning method provided in an embodiment of the present application;
[0029] FIG14 is a flow chart of a positioning method with scene rendering provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0031] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it will be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which the embodiments of this application pertain. The terms used in the embodiments of this application are for the purpose of describing the embodiments of this application only and are not intended to limit this application.
[0032] Before explaining the positioning method of the embodiment of the present application, the professional terms involved in the embodiment of the present application are first explained.
[0033] (1) Mini Program: It is an application that can be used without downloading. Users can open the application by scanning or searching, making the application within reach.
[0034] (2) Indoor navigation: This technology uses AR paths and virtual arrows in indoor spaces, combining AR virtual arrows and directions with indoor maps to guide users to their destinations. This allows users in large buildings to use precise positioning functions to determine their location and find their destination even within the building.
[0035] (3) Indoor visual positioning: It is the process of using laser visual map acquisition equipment to collect the existing indoor environment, establish a three-dimensional point cloud map, and predict the current position and posture of the camera in the indoor environment through a single image taken by the camera.
[0036] (4) AR: It is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time positioning and registration, intelligent interaction, and sensing to simulate computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and then applies them to the real world. The two types of information complement each other, thus achieving an enhancement of the real world.
[0037] (5) VIO: It is an algorithm that integrates camera images and inertial measurement unit (IMU) data to achieve simultaneous positioning and mapping (SLAM). The position and direction are determined by analyzing relevant camera images. Among them, the positioning method of the embodiment of the present application can be applied to the indoor AR navigation solution under the mini program. In the indoor AR navigation solution under the mini program, the VIO under the mini program in the instant messaging application refers to the function open under the mini program visual application programming interface (API). The VIO under the mini program is a visual inertial odometer technology used in the mini program. The VIO under the mini program combines the camera image and sensor data (such as accelerometer and gyroscope) of the mini program, and can estimate the position, speed and direction of the mini program in three-dimensional space in real time. VIO technology can be used for navigation, positioning and control functions of the mini program.
[0038] (6) SLAM: A technology that instructs an autonomous mobile robot to start from an unknown location in an unknown environment, locate its own position and posture by repeatedly observing map features (such as corners, pillars, etc.) during movement, and then incrementally construct a map based on its own position, thereby achieving the purposes of positioning and map construction at the same time.
[0039] (7) IMU: A device that measures the three-axis attitude angle (or angular rate) and acceleration of an object, such as the gyroscope and accelerometer on a mobile phone.
[0040] (8) Structure-From-Motion (SFM): Also known as sparse 3D reconstruction, it is an offline algorithm for 3D reconstruction based on collected unordered images. It is a commonly used technique in computer vision and computer graphics, used to estimate camera motion and scene 3D structure from multiple unordered images. Through SFM technology, the corresponding 3D structure can be recovered from multiple 2D images, thereby achieving scene reconstruction and camera motion estimation. SFM technology has a wide range of applications in augmented reality, virtual reality, computational photography, computer vision and other fields.
[0041] (9) Geographic Coordinate System (GCS): A coordinate system that uses a three-dimensional sphere to define locations on the Earth's surface, so as to reference points on the Earth's surface through longitude and latitude, such as the GCJ02 coordinate system.
[0042] In related technologies, the main process of indoor AR navigation solutions under mini-programs includes: obtaining Bluetooth, WiFi, accelerometer, compass, and gyroscope information through the interface provided by the mini-program or APP; obtaining real-time location information through the integrated positioning engine, updating the location information every second; obtaining the posture information and steering information of the mobile phone; using the real-time positioning results to obtain the surrounding points of interest (POI, Point Of Interest) through the integrated map engine; Interest), signboards and road information; obtain camera photos through a program or APP, and make basic judgments on the mobile phone; when the conditions are met, send the photos to the server for image recognition; among them, when a store and logo appear in the camera photo, the picture changes significantly, and the periodic interval time is reached, it is determined that the conditions are met; then, the positioning system confirms or corrects the current position and direction according to the store name and logo; and, based on road recognition, confirms whether it is currently on the road, and then determines the specific walking road according to the position, and finally confirms the walking direction on the current road according to the compass direction; then, based on the road photo and the phone posture, confirm the relative position of the road in the photo; superimpose AR icons on the photo, and, superimpose POI, logo and other information on the camera in proportion according to the relative position of front, back, left and right; finally, when the road turns, the turning state is confirmed by the gyroscope; and when the turn is in progress, the AR icon will follow the turn change.
[0043] As can be seen from the above description, the relevant positioning technology relies on a server-side integrated positioning engine and mobile phone sensors, namely the mobile phone compass, for positioning. This leads to two problems: First, the server-side integrated positioning engine fails to account for the impact of request transmission latency, making it prone to delayed positioning in areas with poor network connectivity. Furthermore, the image recognition process performed by the server-side integrated positioning engine based on mobile phone images may experience transmission delays in the recognition results, resulting in significant positioning errors. Second, when using the mobile phone compass for positioning, electromagnetic interference in indoor environments or the compass frame rate can affect positioning accuracy. In summary, during the positioning process, transmission delays or sensor accuracy issues can lead to significant positioning errors, resulting in low positioning accuracy and, in turn, poor navigation performance.
[0044] The positioning method provided in an embodiment of the present application first obtains the first posture information of the terminal in response to a positioning request, and then obtains the first virtual posture information of the visual image frame at the current moment in the VIO coordinate system; then, the first posture information and the first virtual posture information are used to generate a transformation matrix; then, the transformation matrix is used to transform the second virtual posture information of the visual image frame at the next moment after the current moment and adjacent to the current moment in the VIO coordinate system to obtain the transformed second virtual posture information; finally, the real posture information of the terminal at the next moment is determined based on the transformed second virtual posture information, thereby realizing the positioning of the terminal between the current moment and the next moment based on the real posture information. In an embodiment of the present application, after obtaining the conversion matrix through the server-side positioning information (i.e., the first pose information) at the current moment and the virtual positioning information in the VIO coordinate system (i.e., the first virtual pose information), during the positioning process, the conversion matrix is directly used to convert the VIO virtual pose information (including position information and orientation) at the next moment, thereby obtaining the real pose information at the next moment, without relying on the server-side positioning engine and terminal sensor to perform positioning at each moment, thereby avoiding the influence of transmission delay and sensor accuracy on positioning accuracy and improving positioning accuracy. In addition, since the conversion matrix at the next moment is determined based on the current moment, the server-side positioning can be performed intermittently during the positioning process to update the conversion matrix, further improving positioning accuracy.
[0045] Here, first, an exemplary application of the positioning device of the embodiment of the present application is described, which is an electronic device for implementing the positioning method. In one implementation, the positioning device (i.e., electronic device) provided in the embodiment of the present application can be implemented as a terminal or as a server. In one implementation, the positioning device provided in the embodiment of the present application can be implemented as any terminal with navigation function and video display function, such as a laptop computer, a tablet computer, a desktop computer, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, an intelligent robot, an intelligent home appliance, and an intelligent vehicle-mounted device; in another implementation, the positioning device provided in the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. Next, an exemplary application when the positioning device is implemented as a server will be described.
[0046] Refer to Figure 1, which is an optional architectural diagram of a positioning system provided in an embodiment of the present application. The embodiment of the present application is explained by taking the positioning method applied to any positioning application as an example. In the embodiment of the present application, the positioning system 10 includes at least a terminal 100, a network 200 and a server 300. Among them, a positioning application can be installed on the terminal 100, and the user can send a positioning request on the client of the positioning application; the server 300 can be a server of the positioning application, and the server 300 can constitute the positioning device of the embodiment of the present application. The terminal 100 is connected to the server 300 through the network 200. The network 200 can be a wide area network or a local area network, or a combination of the two.
[0047] In the embodiment of the present application, a positioning application is running on the terminal 100. When performing positioning processing, the terminal 100 receives the visual image frame at the current moment sent by the user, encapsulates the visual image frame at the current moment into a positioning request, and sends the positioning request to the server 300 through the network 200. In response to the positioning request, the server 300 obtains the first position information of the terminal and obtains the first virtual position information of the visual image frame at the current moment in the VIO coordinate system from the terminal 100; then, using the first position information and the first virtual position information, a transformation matrix is generated; then, using the transformation matrix, the second virtual position information of the visual image frame at the next moment in the VIO coordinate system is transformed to obtain the second virtual position information after the transformation, thereby determining the real position information of the terminal at the next moment based on the second virtual position information after the transformation, thereby realizing the positioning process of the terminal. After determining the second virtual posture information after the conversion process, the server 300 can send the second virtual posture information after the conversion process to the terminal 100 through the network 200. The terminal 100 can display the second virtual posture information after the conversion process on the current interface. Alternatively, the server 300 can determine the real posture information of the terminal at the next moment based on the second virtual posture information after the conversion process, so as to display the real posture information at the next moment on the terminal.
[0048] It should be noted that, since the time interval between the current moment and the next moment in the embodiment of the present application is less than the duration threshold, that is, the time interval between the current moment and the next moment is very short. However, in positioning scenarios, users are usually in a mobile state, or positioning is performed in a navigation scenario, and the positioning method of the embodiment of the present application requires a certain time interval in algorithm implementation. Therefore, the embodiment of the present application no longer uses the current moment's true posture information as the result of the current positioning, but instead determines the terminal's true posture information at the next moment. In this way, positioning errors caused by user movement or movement in a navigation scenario can be avoided.
[0049] In other embodiments, the positioning device can also be implemented as a terminal, that is, a positioning application is installed on the terminal 100, and the positioning method of the embodiment of the present application can be implemented with the terminal as the execution subject. During the implementation process, the terminal 100 obtains the visual image frame collected by the user at the current moment through the positioning application, and in response to the positioning request, obtains the first position information of the terminal from the server 300 through the network 200; then, the terminal 100 obtains the first virtual position information of the visual image frame at the current moment in the VIO coordinate system, and uses the first position information and the first virtual position information to generate a transformation matrix; then, using the transformation matrix, the second virtual position information of the visual image frame at the next moment in the VIO coordinate system is transformed and processed to obtain the second virtual position information after the transformation. After obtaining the second virtual position information after the transformation, the terminal 100 can display the obtained second virtual position information after the transformation on the current interface, or can directly determine the real position information of the terminal at the next moment based on the second virtual position information after the transformation.
[0050] The positioning method provided in the embodiments of the present application can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 can be a cloud server. The cloud server generates a transformation matrix, or uses the transformation matrix to transform the second virtual pose information of the visual image frame at the next moment in the VIO coordinate system to obtain the transformed second virtual pose information.
[0051] In some embodiments, a cloud storage may be provided, and information such as the first pose information and the first virtual pose information may be stored in the cloud storage. Alternatively, the transformation matrix may be stored in the cloud storage. In this way, when the second virtual pose information is subsequently transformed, the stored transformation matrix can be directly selected from the cloud storage for transformation, thereby improving positioning efficiency.
[0052] It's important to note that cloud technology refers to a managed technology that unifies hardware, software, and network resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network, information technology, integration technology, management platform technology, and application technology, all based on the cloud computing business model. It can form a resource pool for on-demand, flexible, and convenient use. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as those for video sites, image sites, and more portals, require significant computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Data of varying levels will be processed separately, and data from various industries will require robust system support, which can be achieved through cloud computing.
[0053] FIG2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device shown in FIG2 may be a positioning device, which includes: at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in the positioning device are coupled together via a bus system 340. It will be understood that the bus system 340 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in FIG2, all various buses are labeled as the bus system 340.
[0054] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0055] The user interface 330 includes one or more output devices 331 that enable presentation of media content, and one or more input devices 332 .
[0056] The memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, and the like. The memory 350 may optionally include one or more storage devices physically located away from the processor 310. The memory 350 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 350 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.
[0057] The operating system 351 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; a network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320. Exemplary network interfaces 320 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB); an input processing module 353 is used to detect one or more user inputs or interactions from one of the one or more input devices 332 and translate the detected inputs or interactions.
[0058] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. FIG2 shows a positioning device 354 stored in a memory 350. The positioning device 354 may be a positioning device in an electronic device. The positioning device 354 may be software in the form of a program or plug-in, and includes the following software modules: a first acquisition module 3541, a second acquisition module 3542, a generation module 3543, a processing module 3544, and a determination module 3545. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.
[0059] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the positioning method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.
[0060] The positioning methods provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the positioning methods in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.
[0061] FIG3 is a schematic flow chart of an optional positioning method provided in an embodiment of the present application. The following will be described in conjunction with the steps shown in FIG3. It should be noted that the positioning method in FIG3 is described by taking a server as an execution subject as an example. As shown in FIG3, the method includes the following steps S101 to S106:
[0062] Step S101: In response to a positioning request, obtain the first position information of the terminal.
[0063] The positioning request includes the visual image frame collected by the terminal at the current moment; the first pose information is the real pose information of the visual image frame at the current moment in the geographic coordinate system.
[0064] Here, the first pose information at least includes the geographical location information and orientation information of the visual image frame at the current moment in the geographical coordinate system.
[0065] In an embodiment of the present application, the visual image frame at the current moment may be the visual image frame at the current moment captured by the terminal when the user is in a different environment, such as the visual image frame at the current moment captured by the terminal when the user is in an indoor building, or the visual image frame at the current moment captured by the terminal when the user is in a scenic area. The terminal can capture the visual image frame at the current moment when the user is in different environments through the camera it carries with it, and render the captured visual image frame at the current moment to display the rendered visual image frame at the current moment on the current interface of the terminal. For example, the terminal can perform Canvas rendering on the captured visual image frame at the current moment, and display the Canvas rendered visual image frame at the current moment on the current interface of the terminal.
[0066] In an embodiment of the present application, a positioning application can be run on the terminal, for example, the positioning application can be run in the form of a mini-program, or the positioning application can be run in the form of an H5 page. The visual image frame at the current moment can be a visual image frame at the current moment collected by the user through the camera operation, or it can be a visual image frame at the current moment automatically collected by the positioning application according to preset conditions. Then, the terminal can encapsulate the visual image frame at the current moment into a positioning request and send it to the server.
[0067] For the first position information, the server can use other electronic devices, such as robots, to collect images of the environment that needs to be positioned in advance; through motion recovery structure (SFM, Structure From Motion) technology, a three-dimensional image point cloud map is established based on the collected images, so that the server can subsequently respond to the positioning request and obtain the first position information of the terminal based on the three-dimensional image point cloud map.
[0068] In an embodiment of the present application, after the server receives the positioning request sent by the terminal, in response to the positioning request, the visual image frame at the current moment can be feature matched with the three-dimensional image point cloud map to obtain the geographical location information and orientation information of the visual image frame at the current moment in the geographic coordinate system. For example, the geographic coordinate system can include the GCJ02 coordinate system, and the server can obtain the geographical location information and orientation information under the GCJ02 coordinate system corresponding to the visual image frame at the current moment. Among them, the GCJ-02 coordinate system uses the WGS-84 ellipsoid, but the longitude and latitude are encrypted to protect security. The GCJ-02 coordinate system is often used in maps, navigation and positioning services. Of course, in some embodiments, the above-mentioned geographic coordinate system can also be the CGCS-2000 coordinate system and the CGCS-2020 coordinate system gradually developed on the basis of GCJ-02, which is not specifically limited in the embodiment of the present application.
[0069] In an embodiment of the present application, the north orientation angle can be used as the reference orientation of the geographic coordinate system, and a specific position in the coordinate system can be used as the reference position of the geographic coordinate system. That is, the server can obtain the orientation information of the visual image frame at the current moment in the GCJ02 coordinate system relative to the reference orientation, as well as the geographical location relative to the reference position, such as the longitude and latitude relative to the reference position.
[0070] In the process of achieving positioning, the server is relied upon to determine the geographic location information and orientation information of the visual image frame at the current moment in the geographic coordinate system, without relying on the terminal's sensor to determine the orientation information. This can avoid the problem of inaccurate orientation information caused by electromagnetic interference or insufficient frame rate of the sensor, thereby improving the accuracy of positioning.
[0071] Step S102: Obtain first virtual pose information of the visual image frame at the current moment in the VIO coordinate system.
[0072] Here, the first virtual posture information includes at least virtual position information and orientation information in the visual image frame VIO coordinate system at the current moment.
[0073] It should be noted that the first virtual posture information is relative position information, that is, the first virtual posture information is the position information and orientation information of the visual image frame at the current moment relative to the reference posture of the VIO coordinate system. The reference posture of the VIO coordinate system can be dynamically changed, and the reference posture of the VIO coordinate system is associated with the position and orientation of the terminal when the server establishes the VIO coordinate system. When the user triggers the VIO function on the terminal, the terminal can obtain the position information and orientation information at the moment the VIO function is triggered, and establish the VIO coordinate system based on the position information and orientation information at the moment the terminal triggers the VIO function as the reference information; then, the terminal determines the first virtual posture information in the VIO coordinate system corresponding to the visual image frame at the current moment based on the reference information of the VIO coordinate system, and sends the first virtual posture information to the server.
[0074] In an embodiment of the present application, the VIO coordinate system may use a rectangular coordinate system to define virtual pose information. For example, when the VIO coordinate system uses a three-dimensional rectangular coordinate system to define virtual pose information, the first virtual pose information may include virtual position information and orientation information corresponding to the X-axis, Y-axis, and Z-axis.
[0075] In the embodiment of the present application, the VIO coordinate system refers to the coordinate system used in VIO. It is understandable that, under normal circumstances, the coordinate systems used in VIO include: a camera coordinate system, which is used to represent the position and attitude of the camera, usually with the optical center of the camera as the origin, the optical axis of the camera as the z-axis, the orientation of the camera as the x-axis, and the plane determined by the z-axis and the x-axis as the y-axis; an inertial measurement unit (IMU) coordinate system: used to represent the output of the accelerometer and gyroscope of the IMU device. The IMU usually contains a three-axis accelerometer and gyroscope. The coordinate system of the accelerometer and gyroscope can be determined according to the installation direction of the device, and usually has a certain deflection with the camera coordinate system; a world coordinate system: used to represent the absolute position and direction of the entire motion system. By fusing the data of the camera and IMU, the motion trajectory of the device and the three-dimensional structure of the scene can be estimated in the world coordinate system.
[0076] In the embodiment of the present application, in the process of determining the first virtual posture information, since the positioning function corresponding to the VIO function has high accuracy, it can return accurate virtual posture information within a certain interval, thereby improving the positioning accuracy of the virtual posture information.
[0077] Step S103: Generate a transformation matrix based on the first pose information and the first virtual pose information.
[0078] Here, the transformation matrix is used to achieve the conversion between the VIO coordinate system and the geographic coordinate system.
[0079] In an embodiment of the present application, the geographic coordinate system defines the real geographic location information and real orientation information corresponding to the visual image frame at the current moment through a spherical surface, and the VIO coordinate system defines the virtual geographic location information and virtual orientation information corresponding to the visual image frame at the current moment through a rectangular coordinate system. Therefore, the server can establish a relationship with the VIO coordinate system based on the reference information of the geographic coordinate system to generate a transformation matrix.
[0080] In an embodiment of the present application, the Cartesian coordinate system can include a rectangular coordinate system and an oblique coordinate system, that is, the Cartesian coordinate system can define the real geographic location information and real orientation information corresponding to the visual image frame at the current moment through the rectangular coordinate system. Therefore, the server can first convert the geographic coordinate system into a Cartesian coordinate system, and then establish the relationship between the Cartesian coordinate system and the VIO coordinate system. For example, the server can perform plane projection processing on the geographic coordinate system based on the reference information of the geographic coordinate system to obtain the geographic coordinate system after plane projection processing; then, adjust the reference information of the geographic coordinate system after plane projection processing to convert the geographic coordinate system after plane projection processing into a Cartesian coordinate system; then, based on the reference information of the Cartesian coordinate system, obtain the second posture information in the Cartesian coordinate system corresponding to the visual image frame at the current moment; and use the second posture information in the Cartesian coordinate system and the first virtual posture information in the VIO coordinate system to determine the conversion matrix.
[0081] It should be noted that the geographic coordinate system may include the GCJ02 coordinate system. Since the GCJ02 coordinate system is usually a coordinate system obtained after deflection processing according to a deflection algorithm, it is impossible to directly convert the GCJ02 coordinate system into a Cartesian coordinate system. In order to achieve the conversion between the geographic coordinate system and the Cartesian coordinate system, the server can convert the GCJ02 coordinate system into the WG584 coordinate system based on the reference information of the GCJ02 coordinate system through a correction algorithm, and perform plane projection processing on the WG584 coordinate system to obtain the WG584 coordinate system after plane projection processing; then, by adjusting the reference information of the WG584 coordinate system after plane projection processing, the WG584 coordinate system after plane projection processing is converted into a Cartesian coordinate system; then, based on the reference information of the Cartesian coordinate system, the second pose information in the Cartesian coordinate system corresponding to the visual image frame at the current moment is obtained, and then the transformation matrix is determined.
[0082] Here, the WG584 coordinate system is a coordinate system used in wireless communications. The WG584 coordinate system is based on the satellite navigation system of the International Telecommunications Satellite Organization (ITU) and uses satellite positioning and pseudo-range measurement technology to obtain coordinate data. The WG584 coordinate system usually uses latitude and longitude in the form of latitude, longitude, and elevation (altitude) to represent the position. The WG584 coordinate system has high accuracy and reliability and is suitable for various wireless communication application scenarios, such as positioning, navigation, communication coverage, signal coverage, etc. In the field of wireless communications, the WG584 coordinate system is widely used in base station positioning, network optimization, signal coverage, and other aspects.
[0083] In some embodiments, in the process of determining the conversion matrix, in order to avoid the cumulative error caused by the posture information, the server can intermittently re-acquire the first posture information and the first virtual posture information, so as to update the conversion matrix based on the re-acquired first posture information and the first virtual posture information to obtain an updated conversion matrix. For example, the server can obtain the first posture information at intervals of a first time length, and the value of the first time length can be set as needed, such as to 5 seconds; the server can also obtain the first virtual posture information at intervals of a second time length, and the value of the second time length can be set as needed, such as to 1 second. In this way, the real-time positioning during the positioning process can be improved, and the positioning accuracy can be guaranteed.
[0084] Step S104: using a transformation matrix to transform the second virtual pose information of the visual image frame at the next moment after the current moment and adjacent to the current moment in the VIO coordinate system to obtain the transformed second virtual pose information.
[0085] The converted second virtual pose information is used to determine the terminal's true pose information at the next moment. Here, the second virtual pose information includes at least virtual position information and orientation information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment.
[0086] In an embodiment of the present application, a positioning application can be run on the terminal, for example, the positioning application can be run in the form of a mini-program, or the positioning application can be run in the form of an H5 page. The visual image frame of the next moment after the current moment and adjacent to the current moment can be the visual image frame of the next moment collected by the user through the camera operation, or it can be the visual image frame of the next moment after the current moment and adjacent to the current moment automatically collected by the positioning application according to preset conditions. After the terminal collects the visual image frame of the next moment, it can send the visual image frame of the next moment to the server in real time, so that the server obtains the second virtual posture information of the visual image frame of the next moment in the VIO coordinate system.
[0087] It should be noted that the second virtual pose information is the position and orientation information of the reference information of the visual image frame at the next moment after and adjacent to the current moment in the VIO coordinate system. In other words, the server obtains the second virtual pose information of the visual image frame at the next moment in the VIO coordinate system based on the reference information of the VIO coordinate system.
[0088] Step S105: determining the real posture information of the terminal at the next moment based on the second virtual posture information after the conversion process.
[0089] In an embodiment of the present application, after the server obtains the second virtual pose information, it can use the transformation matrix to transform the second virtual pose information to obtain the transformed second virtual pose information. For example, if the transformation matrix is used to transform the virtual pose information in the VIO coordinate system into the real pose information in the world coordinate system, the server can directly use the transformation matrix to transform the second virtual pose information; if the transformation matrix is used to transform the real pose information in the world coordinate system into the virtual pose information in the VIO coordinate system, the server can use the inverse matrix of the transformation matrix to transform the second virtual pose information.
[0090] When the second virtual posture information after the conversion process is obtained, the real posture information of the terminal at the next moment can be determined based on the second virtual posture information after the conversion process.
[0091] In an embodiment of the present application, the server can quickly convert the second virtual posture information through the conversion matrix, thereby determining the real posture information of the terminal at the next moment, that is, positioning at the next moment, without the need for feature matching and terminal sensors for positioning. In this way, positioning errors can be reduced and positioning accuracy can be improved.
[0092] In some embodiments, in addition to the above-mentioned positioning scenarios, navigation can also be further performed based on the real position information of the terminal at the next moment determined by the above-mentioned positioning method. That is to say, in the application scenario where the user needs path navigation, the positioning method of the embodiment of the present application can also be applied to the navigation scenario, that is, the path information of this path navigation can also be generated for the user. While positioning the user, the path information can also be displayed to the user to guide the user to the destination corresponding to this path navigation. After obtaining the target position information of this path navigation sent by the terminal, the server determines the path information based on the second virtual position information and the target position information after conversion; the path information includes guidance information for reaching the target position of the target position information. Here, the guidance information includes at least the identification of the target position of the target position information, route guidance and POI information.
[0093] After generating the guidance information, the server can send the guidance information to the terminal; the terminal displays the guidance information in the client of the positioning application to achieve the superposition of the guidance information and the real visual image frame. In this way, the superposition accuracy of the virtual path information and the real image during the positioning process can be effectively improved, making the user's navigation immersion better.
[0094] The positioning method provided in the embodiment of the present application obtains the first pose information of the terminal in response to a positioning request, and obtains the first virtual pose information of the visual image frame at the current moment in the VIO coordinate system, and uses the first pose information and the first virtual pose information to generate a transformation matrix; uses the transformation matrix to transform the second virtual pose information of the visual image frame at the next moment after the current moment and adjacent to the current moment in the VIO coordinate system to obtain the second virtual pose information after the transformation. After obtaining the transformation matrix through the server-side real positioning information and the VIO virtual pose information at the current moment, during the positioning process, the transformation matrix is directly used to transform the VIO virtual pose information at the next moment, thereby obtaining the real pose information at the next moment, without relying on the server-side positioning engine and the terminal sensor to position the next moment, thereby avoiding the influence of transmission delay and sensor accuracy on positioning accuracy, and improving positioning accuracy. In addition, since the transformation matrix of the next moment is determined based on the current moment, the server-side positioning can be intermittently performed during the positioning process to update the transformation matrix, further improving positioning accuracy.
[0095] In some embodiments, the positioning system includes at least a terminal and a server, wherein the terminal is installed with a positioning application that can provide positioning functions. The server is a server of the positioning application, and the positioning method of each embodiment of the present application can be executed through interaction between the server of the positioning application and the terminal.
[0096] FIG4 is another optional flow diagram of the positioning method provided in an embodiment of the present application, wherein FIG4 shows the implementation process of the positioning method implemented by interaction between the terminal and the server, including the following steps S201 to S214:
[0097] Step S201: The terminal collects a visual image frame at the current moment.
[0098] Here, the terminal can collect the visual image frame at the current moment through the camera. The visual image frame at the current moment can be automatically collected by the positioning application according to preset conditions, or collected by the user by clicking a button on the client of the positioning application.
[0099] After collecting the visual image frame at the current moment, the terminal can also render the visual image frame at the current moment to display the rendered visual image frame at the current moment on the current interface of the positioning application for the user to view. For example, the terminal can perform Canvas rendering on the visual image frame at the current moment.
[0100] In step S202 , the terminal encapsulates the collected visual image frame at the current moment into a positioning request.
[0101] Here, the positioning request is used to request positioning of the user's current location.
[0102] The terminal may convert the rendered visual image frame at the current moment into an image in a specific format, and encapsulate the image in the specific format into the positioning request. For example, the terminal may convert the rendered visual image frame at the current moment into an image in JPG format.
[0103] Step S203: The terminal sends a positioning request to the server.
[0104] Step S204: The server obtains the first position information of the terminal in response to the positioning request.
[0105] Here, in the process of the server obtaining the first pose information, the server may make an incorrect recognition due to the existence of similar locations such as floors or elevator entrances, resulting in inaccurate first pose information. In this regard, in an embodiment of the present application, the terminal can also collect auxiliary feature information of the real position corresponding to the visual image frame at the current moment, and send it to the server via a positioning request, so that the server determines the first pose information based on the auxiliary feature information; that is, the positioning request can also include the auxiliary feature information of the real position corresponding to the visual image frame at the current moment, and the auxiliary feature information is used to distinguish similar positions in the visual image frame.
[0106] In some embodiments, referring to FIG5 , FIG5 shows that obtaining the first posture information of the terminal in step S204 can be implemented by following steps S2041 to S2043:
[0107] Step S2041 : In response to the positioning request, determining the auxiliary position information corresponding to the visual image frame at the current moment based on the auxiliary feature information.
[0108] Here, the auxiliary feature information may include at least one of the following: Bluetooth information; WiFi fingerprint information; the auxiliary location information may include information such as floor, elevator entrance, etc.
[0109] When determining auxiliary location information, different locations correspond to different auxiliary feature information. The server can determine the auxiliary location information corresponding to the current visual image frame based on the auxiliary feature information. For example, if the auxiliary feature information includes the first Bluetooth information, the server can determine that the current visual image frame corresponds to the first floor; if the auxiliary feature information includes the second Bluetooth information, the server can determine that the current visual image frame corresponds to the third floor.
[0110] Step S2042: perform feature matching on the visual image frame at the current moment and the preset three-dimensional point cloud map to obtain a matching result.
[0111] Here, the server extracts features from the visual image frame at the current moment, performs feature matching on the extracted features and the preset three-dimensional point cloud map, and obtains a matching result; the matching result includes at least one visual image frame that matches the extracted features, and each visual image frame corresponds to a pose information. Matching the extracted features with the preset three-dimensional point cloud map can be calculating the similarity between the extracted features and the corresponding features in the preset three-dimensional point cloud map. For example, the extracted features can be represented in the form of feature vectors, and the preset three-dimensional point cloud map can also be represented in the form of feature vectors. Therefore, the Euclidean distance between the two feature vectors can be calculated, and the Euclidean distance can be determined as the similarity between the extracted features and the corresponding features in the preset three-dimensional point cloud map.
[0112] Matching an extracted feature with a preset 3D point cloud map means that the similarity between the extracted feature and the corresponding feature in the preset 3D point cloud map is greater than a similarity threshold. For example, the similarity threshold can be 90%. If the calculated similarity is greater than 90%, the extracted feature is considered to match the preset 3D point cloud map.
[0113] Step S2043: Determine the first pose information based on the auxiliary position information and the matching result.
[0114] Here, the server can determine the first position information corresponding to the visual image frame at the current moment by comparing the auxiliary position information with the position information corresponding to at least one visual image frame that matches the extracted features.
[0115] In the embodiment of the present application, the server can effectively improve the problem of low recognition accuracy of the first pose information in similar scenarios by using the auxiliary position information as reference information, thereby improving the positioning accuracy.
[0116] In step S205 , the server obtains first virtual pose information of the visual image frame at the current moment in the VIO coordinate system.
[0117] Here, the server can determine the reference pose of the VIO coordinate system and establish the VIO coordinate system based on the VIO function of the visual API when the user triggers the VIO function; then, for the visual image frame at the current moment, the server can obtain the first virtual pose information relative to the reference pose in the VIO coordinate system based on the VIO function.
[0118] Step S206: The server obtains the third posture information of the terminal.
[0119] Here, the third posture information is the real posture information of the visual image frame at the previous moment before the current moment and adjacent to the current moment in the geographic coordinates.
[0120] In the process of determining the first pose information, the server may also have a large deflection error between the first pose information and the actual pose information. Therefore, in some embodiments, a secondary verification mechanism can be introduced to use the visual image frame of the previous moment before the current moment and adjacent to the current moment and the visual image frame of the current moment to perform mutual verification to improve the accuracy of the first pose information.
[0121] Here, after the server obtains the visual image frame at the previous moment, it can determine the third pose information based on the preset three-dimensional point cloud map and the auxiliary position information corresponding to the visual image frame at the previous moment. In some embodiments, if the server cannot obtain the third pose information of the terminal, that is, positioning fails, the server can obtain the pose information of the visual image frame at the next moment after the current moment and adjacent to the current moment, and use the visual image frame at the next moment and the visual image frame at the current moment to perform mutual verification.
[0122] In step S207, the server uses the first pose information and the third pose information to determine the actual pose deviation.
[0123] Here, the true posture deviation includes the true relative position and true relative orientation of the visual image frame at the previous moment and the visual image frame at the current moment in the world coordinate system.
[0124] In the embodiment of the present application, since the first posture information includes the geographical location information and orientation information of the visual image frame at the current moment in the geographical coordinate system, the third posture information includes the real posture information of the visual image frame at the previous moment in the geographical coordinate system, that is, it includes the real geographical location information and real orientation information of the visual image frame at the previous moment in the geographical coordinate system. Therefore, when determining the real posture deviation, the deviation between the geographical location information of the visual image frame at the current moment in the geographical coordinate system and the real geographical location information of the visual image frame at the previous moment in the geographical coordinate system can be calculated to obtain the real relative position, and the deviation between the orientation information of the visual image frame at the current moment in the geographical coordinate system and the real orientation information of the visual image frame at the previous moment in the geographical coordinate system can be calculated to obtain the real relative orientation.
[0125] Step S208: The server obtains the third virtual pose information of the visual image frame at the previous moment in the VIO coordinate system.
[0126] For the visual image frame at the previous moment, the server can obtain the third virtual pose information relative to the reference pose in the VIO coordinate system based on the VIO function.
[0127] In step S209 , the server determines a virtual posture deviation using the first virtual posture information and the third virtual posture information.
[0128] Here, the virtual posture deviation includes the virtual relative position and virtual relative orientation between the visual image frame at the previous moment and the visual image frame at the current moment in the VIO coordinate system.
[0129] In step S210 , when the real posture deviation matches the virtual posture deviation, the server determines that the verification result is successful.
[0130] In some embodiments, the server may compare the actual posture deviation with the virtual posture deviation, and determine that the verification result is successful if the actual posture deviation and the virtual posture deviation are the same. Alternatively, in some embodiments, the server may also compare the actual posture deviation with the virtual posture deviation, and determine that the verification result is successful if the actual posture deviation and the virtual posture deviation are less than a preset deviation threshold.
[0131] In other embodiments, if the actual pose deviation differs from the virtual pose deviation, the server may determine that the verification result is a verification failure, i.e., there is a positioning error or the VIO coordinate system is offset. In this case, the server can obtain the next visual image frame captured by the terminal and use the current image frame and the next visual image frame to perform a mutual verification until the verification result is a verification success.
[0132] In step S211 , when the verification result is successful, the server generates a transformation matrix using the first pose information and the first virtual pose information.
[0133] In some embodiments, referring to FIG. 6 , FIG. 6 shows that generating the conversion matrix in step S211 can be implemented by following steps S2111 to S2116:
[0134] Step S2111: Acquire first reference information of the geographic coordinate system.
[0135] Here, the first reference information includes a reference orientation and a reference position of the geographic coordinate system.
[0136] Step S2112: performing plane projection processing on the geographic coordinate system based on the first reference information to obtain a geographic coordinate system after plane projection processing.
[0137] The geographic coordinate system defines the geographic location information and orientation information corresponding to the visual image frame through a spherical surface. The range corresponding to the geographic coordinate system is relatively large, so the geographic coordinate system can be plane-projected based on the first reference information to obtain the geographic coordinate system after plane projection processing, so as to reduce the range corresponding to the geographic coordinate system.
[0138] Step S2113: Acquire the second reference information of the geographic coordinate system after plane projection processing.
[0139] Here, the second reference information includes a reference orientation and a reference position of the geographic coordinate system after plane projection processing.
[0140] Step S2114: converting the geographic coordinate system after the plane projection processing into a Cartesian coordinate system by adjusting the second reference information.
[0141] Since the VIO coordinate system defines the geographic location information and orientation information corresponding to the visual image frame through a rectangular coordinate system, which is different from the definition method of the world coordinate system, in order to establish the relationship between the geographic coordinate system after plane projection processing and the VIO coordinate system, the server can convert the geographic coordinate system after plane projection processing into a Cartesian coordinate system. The Cartesian coordinate system can define the geographic location information and orientation information corresponding to the visual image frame through a rectangular coordinate system, and the reference information of the Cartesian coordinate system is the same.
[0142] Here, after determining the reference information of the Cartesian coordinate system, the server may adjust the second reference information so that the second reference information is consistent with the reference information of the Cartesian coordinate system, so as to transform the geographic coordinate system after plane projection processing into the Cartesian coordinate system.
[0143] Step S2115: Acquire the third reference information of the Cartesian coordinate system.
[0144] Step S2116: Based on the third reference information, the first pose information is converted into second pose information.
[0145] Here, based on the third reference information of the Cartesian coordinate system, the server can convert the posture information of the visual image frame at the current moment in the visual coordinate system relative to the first reference information (i.e., the first posture information) into the posture information of the visual image frame at the current moment in the Cartesian coordinate system relative to the third reference information (i.e., the second posture information).
[0146] Step S2117: Generate a transformation matrix using the second pose information and the first virtual pose information.
[0147] Since the Cartesian coordinate system and the VIO coordinate system have the same definition, for the same visual image frame at the current moment, the server can obtain the transformation matrix T by determining the rotation relationship between the second pose information and the first virtual pose.
[0148] In step S212, the server uses the transformation matrix to transform the second virtual pose information in the VIO coordinate system corresponding to the visual image frame at the next moment to obtain the transformed second virtual pose information.
[0149] Here, the second virtual posture information after the conversion process includes the position information and orientation information in the geographic coordinate system corresponding to the visual image frame at the next moment.
[0150] For the next moment after the current moment and adjacent to the current moment, the terminal can use the camera to capture the visual image frame of the next moment. The visual image frame of the next moment can be automatically captured by the positioning application according to preset conditions, or it can be captured by the positioning application server in response to the user's click of a button on the positioning application client. The terminal can then render the visual image frame of the next moment and display it on the current interface of the positioning application for the user to view.
[0151] In addition, the terminal can also send the captured visual image frame at the next moment to the server, so that the server can obtain the second virtual pose information of the visual image frame at the next moment in the VIO coordinate system relative to the reference pose based on the VIO function. The server then uses the transformation matrix to transform the second virtual pose information to determine the terminal's true pose information at the next moment, without having to rely on auxiliary positions and 3D point cloud maps for positioning at the next moment. This avoids the impact of transmission delay and sensor accuracy on positioning accuracy, thereby improving positioning accuracy.
[0152] In some embodiments, in application scenarios where path navigation is required, the user can also input the target location information of this path navigation through the positioning application on the terminal, and the input method includes but is not limited to text input and voice input; after the server receives the target location information of this path navigation sent by the terminal, it determines the path information based on the converted second virtual posture information and target location information; the path information includes guidance information for reaching the target location corresponding to the target location information.
[0153] Here, the target location information can be the target location in a specific environment that the user needs to reach when in a specific environment. For example, the target location information can be the target location in an indoor building that the user needs to reach when in an indoor building, or it can be the target location in a scenic area that the user needs to reach when in a scenic area.
[0154] In an embodiment of the present application, during the implementation process of determining the path information, the server obtains the model information corresponding to the visual image frame at the current moment; then, based on the model information corresponding to the visual image frame at the current moment, a virtual scene in the VIO coordinate system is generated; then, based on the first pose information and the target position information, guidance information for reaching the target position from the position corresponding to the first pose information is determined; and through the transformation matrix, the guidance information for reaching the target position from the position corresponding to the first pose information is fused into the virtual scene to obtain a fused virtual scene; finally, based on the second virtual pose information after the conversion processing, guidance information for reaching the target position corresponding to the target position information is determined from the fused virtual scene.
[0155] In computer vision and computer graphics, the model information corresponding to a visual image frame is the 3D geometric model obtained by mapping the pixel data in the image frame onto a geometric model in 3D space. The model information corresponding to the current visual image frame includes information such as the POI, object, and scene in the current visual image frame in the geographic coordinate system.
[0156] In an embodiment of the present application, the process in which the server obtains the model information corresponding to the visual image frame at the current moment involves recovering or inferring the information of the three-dimensional structure from the two-dimensional image, that is, the so-called "conversion from image to model". This model can be the surface, shape or more abstract geometric structure of an object in a geographic coordinate system. Obtaining model information from a visual image frame can be achieved through the following steps: First, detecting and extracting feature points of POIs, objects, scenes, etc. in the visual image frame, such as corners, edges, textures, etc. These feature points will be used to estimate the shape and position of the object in subsequent steps. Then, determine the intrinsic parameters (such as focal length, point, etc.) and extrinsic parameters (such as the posture and position of the camera) of the camera, in order to convert the image coordinates into the geographic coordinate system of the scene. Next, estimate the three-dimensional coordinates of the object surface. Then, reconstruct the complete three-dimensional model of the object.
[0157] The virtual scene can be a three-dimensional virtual scene, which can provide users with POIs, guide signs or guide paths with different distances and sizes, thereby enhancing the user's immersion and sense of reality during the path navigation and positioning process.
[0158] In some embodiments, the server can also initialize and generate a virtual scene in the VIO coordinate system based on the reference information of the VIO coordinate system; then, use the transformation matrix to transform the model information corresponding to the visual image frame at the current moment to obtain the model information of the visual image frame at the current moment in the VIO coordinate system, and fuse the model information in the VIO coordinate system corresponding to the visual image frame at the current moment into the virtual scene; then, the server can use the transformation matrix to fuse the guidance information used to reach the target position from the position corresponding to the first pose information into the virtual scene to obtain the fused virtual scene; finally, for the next moment after the current moment and adjacent to the current moment, determine the guidance information used to reach the target position corresponding to the target position information from the fused virtual scene.
[0159] In the above process, the model information corresponding to the visual image frame may change as time changes, that is, the user's visual pose information in the world coordinate system may change; the visual pose information may include visual position and visual orientation, such as camera position and rotation angle; in order to ensure the accuracy of the superposition of guidance information and real visual image frames, it is necessary to ensure that the visual pose information in the virtual scene is consistent with the user's visual pose information in the world coordinate system.
[0160] In some embodiments, the terminal can obtain model information corresponding to the visual image frame of the next moment after the current moment and adjacent to the current moment; when the model information corresponding to the visual image frame of the current moment is different from the model information corresponding to the visual image frame of the next moment, it can be determined that the visual pose information of the visual image frame of the next moment in the world coordinate system has changed; then, based on the model information corresponding to the visual image frame of the next moment, the visual pose information corresponding to the fused virtual scene can be updated to obtain an updated virtual scene; finally, based on the second virtual pose information after the conversion processing, the guidance information for reaching the target position corresponding to the target position information is determined from the updated virtual scene. In other words, by detecting the changes in the model information corresponding to the visual image frames at different moments, the visual pose information in the virtual scene is updated in real time, so that the pose information of the guidance information is consistent with the pose information of the real visual image frame.
[0161] In the embodiment of the present application, by acquiring the visual posture information of the visual image frame and updating the visual posture information of the virtual scene in real time, the accuracy of the superposition of the virtual guidance information and the real visual image frame can be guaranteed, thereby improving the navigation effect.
[0162] In step S213, the server sends the converted second virtual posture information to the terminal.
[0163] In some embodiments, the server may further send guidance information for reaching the target location corresponding to the target location information to the terminal.
[0164] In step S214, the terminal displays the converted second virtual posture information on the current interface.
[0165] In some embodiments, the terminal may further display guidance information for reaching the target location corresponding to the target location information on the current interface.
[0166] The positioning method provided in the embodiment of the present application, in response to the positioning request, obtains the first pose information of the terminal, and then obtains the first virtual pose information of the visual image frame at the current moment in the VIO coordinate system; then, the first pose information and the first virtual pose information are used to generate a conversion matrix; finally, the conversion matrix is used to convert the second virtual pose information in the VIO coordinate system corresponding to the visual image frame at the next moment after the current moment and adjacent to the current moment, to obtain the second virtual pose information after the conversion. In the positioning process, the conversion matrix is first obtained by the real positioning information of the server at the current moment and the VIO virtual positioning information, and then the conversion matrix is used to convert the VIO virtual pose information (including position information and orientation) at the next moment to obtain the real pose information at the next moment, without relying on the server-side positioning engine and the terminal sensor to position the next moment, thereby avoiding the influence of transmission delay and sensor accuracy on positioning accuracy and improving positioning accuracy. In addition, since the conversion matrix of the next moment is determined based on the current moment, the server positioning can be performed intermittently during the positioning process to update the conversion matrix, further improving positioning accuracy.
[0167] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.
[0168] Current AR navigation solutions for mini-programs combine virtual and real elements by simply overlaying POIs and signs as icons on the camera image in proportion to their front-to-back and left-to-right relative positions. This doesn't achieve the same rendering of a three-dimensional scene as the actual camera image. Furthermore, during positioning, while the server-side integrated positioning engine returns location information, azimuth angle information relies on the phone's compass, which presents two issues:
[0169] On the one hand, electromagnetic interference may exist in indoor environments, which will affect the direction of the mobile phone compass and cause inaccurate angular information to be obtained. At the same time, the frame rate returned by the compass in the mini program is insufficient, which will also cause inaccurate angular information to be obtained when the mobile phone moves quickly. On the other hand, during positioning through the integrated positioning engine on the server side, the impact of the round-trip transmission time when sending the positioning request to the server is not considered, resulting in the possibility of delayed positioning in areas with poor network conditions. In addition, when sending the acquired camera image to the server for image recognition, the possible network delay during the image transmission process and the possibility that the current orientation and displacement of the mobile phone may have changed are not considered. Combined with the above factors, there will be a mismatch between the virtual information and the actual position and orientation of the camera in the real world, resulting in low AR navigation accuracy and poor user immersion experience.
[0170] An embodiment of the present application provides a positioning method, which is applied to indoor AR walking navigation scenarios. As shown in FIG7A , when performing indoor AR navigation under a mini program, in the visual positioning process 71, the current camera image is obtained through the mini program camera API and transmitted to the visual positioning server for the first visual positioning, that is, the accurate and timely relative position is determined by the mini program visual odometer, for example, the geographic location is determined and the camera reference point is established. Then, in the relative position positioning process 72, the mini program side VIO is used for relative position positioning, that is, the geographic reference point conversion relationship matrix is established through the visual positioning server. Subsequently, the geographic reference point conversion relationship matrix is used to convert the current camera pose matrix returned by the mini program side VIO into geographic coordinates for positioning, and an accurate geographic location is obtained, and virtual information is used to indicate that the user has reached the destination. In addition, in the similar area recognition process 73, by introducing Bluetooth or WIFI positioning services, problems such as cross-layer and similar area positioning errors of pure visual positioning solutions can also be improved, for example, auxiliary judgment of floors or areas can be achieved. This effectively improves the large positioning reference point errors caused by sensor accuracy and transmission latency issues in traditional mini-program environments, resulting in a mismatch between real-world and virtual information, and poor AR realism and immersion. This improves positioning accuracy, improves the accuracy of the overlay of virtual information with the camera's real image, and provides greater real-time intermediate positioning, resulting in a better user immersion experience. Furthermore, the introduction of Bluetooth and Wi-Fi positioning services can also improve cross-layer and similar area positioning errors in pure visual positioning solutions, further improving positioning accuracy.
[0171] The implementation process of the positioning method provided in the embodiment of the present application, as shown in FIG7B , includes the following steps S301 to S307:
[0172] In step S301, the mini program uses a robot to collect images of the building that needs AR navigation, and then uses SFM technology to process the collected images into camera visual image frames to establish a three-dimensional image point cloud map. Based on image retrieval and feature matching visual positioning technology, a visual positioning server is built.
[0173] In step S302, the mini program collects Wi-Fi fingerprints or Bluetooth information in the building.
[0174] In this way, the floor and area can be judged later through the WIFI fingerprint or Bluetooth information collected on the mini program.
[0175] Step S303: Use the VIO capability of the visual API on the mini-program side to initialize a three-dimensional rendering scene consistent with the VIO coordinate system, and render the visual image frame into the three-dimensional rendering scene.
[0176] Step S304: During the first positioning, the mini program sends the visual image frame at the current moment to the visual positioning server for visual positioning. After the visual positioning server performs image retrieval and feature matching based on the visual image frame at the current moment, it returns the pose information (including geographic location and orientation information) of the visual image frame at the current moment to the mini program. The mini program adopts a two-request verification mechanism to eliminate the situation where the deflection error is large (i.e., the direction of the virtual visual return point is significantly different from the actual camera direction).
[0177] FIG8 is a flow chart of the first visual positioning process provided by an embodiment of the present application. As shown in FIG8 , first, the mini program side obtains the visual image frame at the current moment (i.e., an image frame in the frame image stream at the current moment), and performs Canvas rendering processing 801 on the visual image frame at the current moment, obtains the visual image frame at the current moment after Canvas rendering processing, and displays it on the current interface of the terminal; then, the mini program side converts the visual image frame at the current moment after Canvas rendering processing into an image in JPG format 802, encapsulates it into a positioning request, and sends it to the visual positioning server; After receiving the positioning request, the visual positioning server returns the pose information in the GCJ02 coordinate system corresponding to the visual image frame at the current moment to the mini program end based on image retrieval and feature matching, so that the mini program end can subsequently calculate the transformation matrix 803 based on the pose information in the GCJ02 coordinate system corresponding to the visual image frame at the current moment; wherein, the pose information in the GCJ02 coordinate system corresponding to the visual image frame at the current moment includes the GCJ02 coordinates compared to the north orientation angle (i.e., the reference orientation information); the transformation matrix 803 is used to realize the conversion between the VIO coordinate system and the GCJ02 coordinate system.
[0178] During the first visual positioning process, there may be a situation where the direction returned by the visual positioning server is significantly different from the actual direction of the camera, resulting in a positioning error. In response to this, the embodiment of the present application adds a secondary verification process, using two requests for mutual verification during the first positioning, thereby improving the accuracy of the first positioning. First, for the first visual image frame corresponding to the current moment, after the applet side receives the response (Response) that the first visual image frame is successfully positioned, it immediately or at a short interval sends a positioning request to the visual positioning server again to verify the second visual image frame corresponding to the next moment; then, after the applet side receives the response that the second visual image frame is successfully positioned, it verifies whether the posture information returned by the second visual image frame is consistent with the posture information returned by the first visual image frame and the position information (such as longitude and latitude) and direction of the first visual image frame and the second visual image frame obtained based on the VIO function; if they are inconsistent, it means that there is a positioning error or that the VIO function is unstable. The third visual image frame and the second visual image frame can be verified in the above manner, or the first visual positioning process can be re-executed; if they are consistent, it means that the verification is successful, and step S305 can be executed. If the mini program does not receive a successful response to the second visual image frame, the mini program can send a positioning request to the visual positioning server again until it receives a successful positioning response message from the visual positioning server and successfully verifies the position with the first visual image frame. If the verification is successful, the mini program will carry the context information of the pose information returned by the second visual image frame when sending a positioning request to the visual positioning server next time, so that the visual positioning server can determine that the positioning is continuous.
[0179] In addition, when using the visual positioning server for positioning, there may be errors in the recognition of similar scenes such as cross-layers or elevator entrances, resulting in inaccurate positioning. To this end, the embodiment of the present application has added a positioning process for similar scenes such as cross-layers and elevator entrances. After the mini program collects the Bluetooth and WIFI information of the building that needs AR navigation, the collected information is sent to the visual positioning server as auxiliary parameters, so that the visual positioning server performs positioning based on the auxiliary parameters and returns the posture information. In this way, the problem of incorrect recognition in similar scenes can be effectively improved, and the positioning accuracy can be improved.
[0180] In step S305, based on the VIO capability of the visual API of the mini-program side, the mini-program side locates the virtual posture of the visual image frame at the current moment, and establishes a relative position geographic reference point conversion relationship matrix through the first positioning, so that the conversion relationship matrix can be used to convert the posture information corresponding to the visual image frame at subsequent moments into posture information under geographic coordinates for positioning and navigation; wherein, VIO can return a relatively accurate relative position within a certain interval.
[0181] Figure 9 is a schematic diagram of the VIO coordinate system and the geographic coordinate system provided in an embodiment of the present application. As shown in Figure 9, the geographic coordinate system defines the real posture information corresponding to the visual image frame at the current moment through a spherical surface, and the VIO coordinate system defines the information corresponding to the visual image frame at the current moment through a rectangular coordinate system. Therefore, it is necessary to calculate the transformation matrix to realize the conversion between the geographic coordinate system and the VIO coordinate system.
[0182] FIG10 is a flow chart of the calculation transformation matrix provided by an embodiment of the present application. As shown in FIG10 , after the mini-program end determines the reference information 1001 of the GCJ02 coordinate system, the GCJ02 coordinate system can be converted into the WG584 geographic coordinate system 1002 through the correction algorithm; by performing plane projection processing on the WG584 geographic coordinate system, the WG584 geographic coordinate system 1003 after plane projection processing is obtained; based on the reference information of the WG584 geographic coordinate system after plane projection processing, the position information under the local Cartesian coordinate system 1004 corresponding to the visual image frame at the current moment is obtained; combined with the position information under the VIO coordinate system 1005 corresponding to the visual image frame at the current moment, the transformation matrix can be calculated. In addition, after a certain time interval (for example, 5 seconds), the mini-program end can resend a positioning request to the server side. At the same time, the mini-program end can perform VIO relative position positioning every 1 second to update the transformation matrix.
[0183] In step S306, the mini program obtains a coordinate point string corresponding to the target location based on the route planning service, and draws the coordinate point string as guidance information into the three-dimensional rendering scene.
[0184] In an embodiment of the present application, a rendering process combining a 3D scene with a real camera image can be incorporated to enhance user immersion. Based on the route planning service, the mini-program combines the current visual image frame and the target location of this path navigation to obtain a coordinate point string in the geographic coordinate system corresponding to the target location. The coordinate point string is then rendered into the 3D rendered scene as guidance information using the transformation matrix. This guidance information includes route guidance, signs, PIO, and other virtual guidance information.
[0185] Figure 11 is a flow chart of the scene rendering process 1100 provided in an embodiment of the present application. As shown in Figure 11, after the mini-program end determines the reference information of the GCJ02 coordinate system, the GCJ02 coordinate system is converted into the WG584 geographic coordinate system through a correction algorithm; the WG584 plane projection coordinate system is obtained by plane projection processing of the WG584 geographic coordinate system; based on the reference information of the WG584 plane projection coordinate system, the WG584 plane projection coordinate system is converted into a Cartesian coordinate system; using the transformation matrix, the coordinate point string in the Cartesian coordinate system is drawn as guidance information into the three-dimensional rendering scene in the VIO coordinate system.
[0186] In step S307, the mini program detects changes in the VIO view matrix and updates the visual pose information (i.e., camera coordinates and orientation angle) in the 3D rendering scene in real time.
[0187] In an embodiment of the present application, the model information contained in the visual image frames at different moments may be different. The mini-program end can detect the model information contained in the visual image frame at the current moment and the visual image frame at the next moment after the current moment and adjacent to the current moment, thereby determining whether the VIO view matrix has changed. In the case that the VIO view matrix changes, the camera coordinates and orientation in the three-dimensional rendering scene are updated based on the model information contained in the visual image frame at the next moment after the current moment and adjacent to the current moment, so that the guidance information can match the visual image frame at the next moment after the current moment and adjacent to the current moment, thereby ensuring the accuracy of the superposition of the guidance information and the visual image frame at the next moment after the current moment and adjacent to the current moment.
[0188] Steps S304 to S307 are executed in a loop until the target location of this route navigation is reached.
[0189] In an embodiment of the present application, when the user moves toward the target position based on the guidance information, the mini program end can detect the second virtual posture information of the visual image frame at the next moment after the current moment and adjacent to the current moment in the VIO coordinate system; the second virtual posture information is transformed based on the transformation matrix to obtain the second virtual posture information after the transformation; based on the second virtual posture information after the transformation and the target position information, it can be determined whether the user has reached the target position.
[0190] Figure 12 is a flow chart of the navigation process 1200 provided in an embodiment of the present application. As shown in Figure 13, the mini program end uses the inverse matrix of the transformation matrix to transform the VIO coordinate system to obtain a Cartesian coordinate system; then, by adjusting the reference information of the local Cartesian coordinate system, the Cartesian coordinates are transformed into the WG584 plane projection coordinate system; then, through the reverse projection algorithm, the WG584 plane projection coordinate system is reversely projected to obtain the WG584 geographic coordinate system; finally, through the deflection algorithm, the WG584 geographic coordinate system is transformed into the GCJ02 coordinate system to obtain the second virtual posture information after the transformation processing in the GCJ02 coordinate system.
[0191] Figure 13 is a schematic diagram of the application scenario of the positioning method provided in an embodiment of the present application. As shown in Figure 13, during the process of navigating from the user's current location (F1) to the target location (F3), guidance signs, guidance paths and other content can be displayed in the visual image frame at the current moment to generate path information for this path navigation for the user.
[0192] Figure 14 is a flow chart of the positioning method with scene rendering provided by an embodiment of the present application. As shown in Figure 14, first, according to the first positioning process 141, the processed visual image frame is sent to the server to request the pose information of the visual image frame at the current moment. Then, after the pose information is verified, a transformation matrix is established based on the verified pose information and the pose matrix in the VIO coordinate system. In this way, the situation where the pose information has a large deflection error can be eliminated, thereby improving the accuracy of the first positioning. In the positioning process 142 on the mini-program side, the mini-program side determines the virtual geographic coordinates and orientation of the visual image frame at the subsequent moment in the VIO coordinate, and transforms the virtual geographic coordinates and orientation of the visual image frame at the subsequent moment in the VIO coordinate through the transformation matrix to perform positioning and generate path information to the target location for the user.
[0193] In an embodiment of the present application, in order to increase the realism and immersion of the user's AR navigation, a rendering initialization process 143 and a three-dimensional scene update process 144 are added in the process of generating path information. After the mini program generates a three-dimensional scene based on the visual image frame, the visual posture information in the three-dimensional scene is updated in real time based on the view matrix corresponding to the visual image frame at subsequent moments, so that the user can perceive the differences in distance or size of the indication signs and indication paths in different positions, thereby improving the immersion and realism of the user's AR navigation.
[0194] It is understandable that in the embodiments of the present application, the content involving user information, such as visual image frames and target location information collected by the terminal, if it involves data related to user information or corporate information, when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0195] The following continues to describe an exemplary structure of the positioning device 354 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in Figure 2, the positioning device 354 includes: a first acquisition module 3541, configured to obtain the first posture information of the terminal in response to a positioning request; the positioning request includes the visual image frame at the current moment collected by the terminal; the first posture information is the real posture information of the visual image frame at the current moment in the geographic coordinate system; a second acquisition module 3542, configured to obtain the first virtual posture information of the visual image frame at the current moment in the VIO coordinate system; a generation module 3543, configured to generate a transformation matrix based on the first posture information and the first virtual posture information; a processing module 3544, configured to use the transformation matrix to transform the second virtual posture information of the visual image frame at the next moment after the current moment and adjacent to the current moment in the VIO coordinate system to obtain the transformed second virtual posture information; a determination module 3545, configured to determine the real posture information of the terminal at the next moment based on the transformed second virtual posture information.
[0196] In some embodiments, the generation module 3543 is further configured to obtain first reference information of the geographic coordinate system; the first reference information includes reference orientation information and reference position information of the geographic coordinate system; based on the first reference information, the first posture information is converted into second posture information; the second posture information is the real posture information of the visual image frame at the current moment in the Cartesian coordinate system; based on the second posture information and the first virtual posture information, the transformation matrix is generated.
[0197] In some embodiments, the generation module 3543 is further configured to perform plane projection processing on the geographic coordinate system based on the first reference information to obtain the geographic coordinate system after plane projection processing; obtain second reference information of the geographic coordinate system after plane projection processing; the second reference information includes the reference orientation information and reference position information of the geographic coordinate system after plane projection processing; by adjusting the second reference information, the geographic coordinate system after plane projection processing is converted into a Cartesian coordinate system; obtain third reference information of the Cartesian coordinate system; the third reference information includes the reference orientation information and reference position information of the Cartesian coordinate system; based on the third reference information, the first posture information is converted into the second posture information.
[0198] In some embodiments, the generation module 3543 is further configured to obtain third posture information of the terminal; the third posture information is the real posture information of the visual image frame at the previous moment before the current moment and adjacent to the current moment under the geographic coordinates; the third posture information and the first posture information are verified to obtain a verification result; when the verification result is successful, the transformation matrix is generated based on the first posture information and the first virtual posture information.
[0199] In some embodiments, the generation module 3543 is further configured to determine the real posture deviation based on the first posture information and the third posture information; obtain the third virtual posture information of the visual image frame of the previous moment before the current moment and adjacent to the current moment in the VIO coordinate system; use the first virtual posture information and the third virtual posture information to determine the virtual posture deviation; when the real posture deviation is the same as the virtual posture deviation, determine that the verification result is a successful verification.
[0200] In some embodiments, the device also includes: a path generation module, configured to receive the target position information of this positioning sent by the terminal; determine the path information based on the second virtual posture information after the conversion processing and the target position information; the path information includes guidance information to reach the target position corresponding to the target position information.
[0201] In some embodiments, the path generation module is further configured to obtain model information of the visual image frame at the current moment; generate a virtual scene in the VIO coordinate system based on the model information of the visual image frame at the current moment; determine guidance information for reaching the target position from the position corresponding to the first posture information based on the first posture information and the target position information; fuse the guidance information for reaching the target position from the position corresponding to the first posture information into the virtual scene through the transformation matrix to obtain a fused virtual scene; and determine guidance information for reaching the target position from the fused virtual scene based on the second virtual posture information after the conversion processing.
[0202] In some embodiments, the path generation module is further configured to obtain model information of the visual image frame at the next moment after the current moment and adjacent to the current moment; when the model information of the visual image frame at the current moment is different from the model information corresponding to the visual image frame at the next moment, the visual pose information of the fused virtual scene is updated based on the model information of the visual image frame at the next moment to obtain an updated virtual scene; based on the second virtual pose information after the conversion processing, the guidance information for reaching the target position is determined from the updated virtual scene.
[0203] In some embodiments, the positioning request also includes auxiliary feature information of the real position of the visual image frame at the current moment; the auxiliary feature information is used to distinguish similar positions in the visual image frame; the first acquisition module is also configured to respond to the positioning request and determine the auxiliary position information of the visual image frame at the current moment based on the auxiliary feature information; perform feature matching on the visual image frame at the current moment with a preset three-dimensional point cloud map to obtain a matching result; and determine the first posture information based on the auxiliary position information and the matching result.
[0204] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.
[0205] The present invention provides a computer program product comprising executable instructions, which are computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method described in the present invention.
[0206] An embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, the method shown in Figure 3.
[0207] In some embodiments, the storage medium can be a computer-readable storage medium, such as a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.
[0208] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0209] By way of example, executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). By way of example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected by a communication network.
[0210] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.
Claims
1. A positioning method, the method being performed by an electronic device, the method comprising: In response to the positioning request, obtaining the first position information of the terminal; The positioning request includes a visual image frame at the current moment collected by the terminal; The first posture information is the real posture information of the visual image frame at the current moment in the geographic coordinate system; Obtaining first virtual pose information of the visual image frame at the current moment in a visual inertial odometer (VIO) coordinate system; Generate a transformation matrix based on the first posture information and the first virtual posture information; Using the transformation matrix, transform the second virtual pose information of the visual image frame at the next moment after the current moment and adjacent to the current moment in the VIO coordinate system to obtain the transformed second virtual pose information; The real posture information of the terminal at the next moment is determined based on the second virtual posture information after the conversion process.
2. The method according to claim 1, wherein: The generating a transformation matrix based on the first posture information and the first virtual posture information includes: Acquire first reference information of the geographic coordinate system; the first reference information includes reference orientation information and reference position information of the geographic coordinate system; Based on the first reference information, the first posture information is converted into second posture information; the second posture information is the real posture information of the visual image frame at the current moment in a Cartesian coordinate system; The transformation matrix is generated based on the second pose information and the first virtual pose information.
3. The method according to claim 2, wherein: The converting the first posture information into second posture information based on the first reference information includes: Performing plane projection processing on the geographic coordinate system based on the first reference information to obtain a geographic coordinate system after plane projection processing; Acquire second reference information of the geographic coordinate system after the plane projection processing; the second reference information includes reference orientation information and reference position information of the geographic coordinate system after the plane projection processing; By adjusting the second reference information, the geographic coordinate system after the plane projection processing is converted into a Cartesian coordinate system; Acquire third reference information of the Cartesian coordinate system; the third reference information includes reference orientation information and reference position information of the Cartesian coordinate system; Based on the third reference information, the first posture information is converted into the second posture information.
4. The method according to any one of claims 1 to 3, wherein: The generating a transformation matrix based on the first posture information and the first virtual posture information includes: Acquire third posture information of the terminal; the third posture information is the real posture information of the visual image frame at a previous moment before the current moment and adjacent to the current moment under the geographic coordinates; Performing verification processing on the third posture information and the first posture information to obtain a verification result; When the verification result is successful, the transformation matrix is generated based on the first posture information and the first virtual posture information.
5. The method according to claim 4, wherein: The verifying the third posture information and the first posture information to obtain a verification result includes: Determining a true posture deviation based on the first posture information and the third posture information; Acquire third virtual pose information of a visual image frame at a previous moment before the current moment and adjacent to the current moment in the VIO coordinate system; Determine a virtual posture deviation using the first virtual posture information and the third virtual posture information; When the real posture deviation is the same as the virtual posture deviation, the verification result is determined to be a successful verification.
6. The method according to any one of claims 1 to 5, wherein: The method further comprises: Receiving target location information of this route navigation sent by the terminal; Based on the converted second virtual posture information and the target position information, path information is determined; the path information includes guidance information for reaching the target position corresponding to the target position information.
7. The method according to claim 6, wherein: The determining of path information based on the converted second virtual posture information and the target position information includes: Acquire model information of the visual image frame at the current moment; Generate a virtual scene in a VIO coordinate system based on the model information of the visual image frame at the current moment; Based on the first posture information and the target position information, determining guidance information for reaching the target position from the position corresponding to the first posture information; By means of the transformation matrix, the guidance information for reaching the target position from the position corresponding to the first posture information is integrated into the virtual scene to obtain a integrated virtual scene; Based on the converted second virtual pose information, guidance information for reaching the target position is determined from the fused virtual scene.
8. The method according to claim 7, wherein: The step of determining guidance information for reaching the target position from the fused virtual scene based on the converted second virtual posture information includes: Acquire model information of a visual image frame at a next moment after the current moment and adjacent to the current moment; When the model information of the visual image frame at the current moment is different from the model information corresponding to the visual image frame at the next moment, updating the visual pose information of the fused virtual scene based on the model information of the visual image frame at the next moment to obtain an updated virtual scene; Based on the converted second virtual posture information, guidance information for reaching the target position is determined from the updated virtual scene.
9. The method according to any one of claims 1 to 8, wherein: The positioning request also includes auxiliary feature information of the real position of the visual image frame at the current moment; The auxiliary feature information is used to distinguish similar locations in the visual image frame; The step of obtaining the first position information of the terminal in response to the positioning request includes: In response to the positioning request, determining auxiliary position information of the visual image frame at the current moment based on the auxiliary feature information; Perform feature matching between the visual image frame at the current moment and the preset three-dimensional point cloud map to obtain a matching result; Based on the auxiliary position information and the matching result, the first posture information is determined.
10. A positioning device, comprising: A first acquisition module, configured to acquire the first position information of the terminal in response to a positioning request; The positioning request includes a visual image frame at the current moment collected by the terminal; The first posture information is the real posture information of the visual image frame at the current moment in the geographic coordinate system; A second acquisition module is configured to acquire first virtual pose information of the visual image frame at the current moment in the VIO coordinate system; A generating module, configured to generate a transformation matrix based on the first posture information and the first virtual posture information; a processing module configured to transform the second virtual pose information of the visual image frame at a next moment after the current moment and adjacent to the current moment in the VIO coordinate system using the transformation matrix to obtain the transformed second virtual pose information; A determination module is configured to determine the terminal in the next step based on the second virtual posture information after the conversion process. The real pose information at a moment.
11. An electronic device, comprising: A memory for storing executable instructions; A processor, configured to implement the positioning method according to any one of claims 1 to 9 when executing the executable instructions stored in the memory.
12. A computer-readable storage medium storing executable instructions, for causing a processor to execute the executable instructions to implement the positioning method according to any one of claims 1 to 9.
13. A computer program product, the computer program product comprising executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device executes the positioning method as described in any one of claims 1 to 9.
Citation Information
Patent Citations
Navigation method and device for unmanned equipment, storage medium and unmanned equipment
CN111238486A
Pose determination method and device, electronic equipment and storage medium
CN113074726A
Method and device for rendering image, electronic equipment and storage medium
CN115100342A
Lightweight pose measurement method fused with inertial measurement unit scale information
CN116086444A
Enhanced content positioning
US11625857B1
Cited By
Robot multi-source data fusion positioning method and device and computer equipment
CN120800354A
