Augmented Reality Method Combining GNSS Data and Augmented Reality Device

Abstract

The present invention provides an augmented reality method and an augmented reality device that incorporate GNSS data. Among them, the augmented reality method includes: setting a plurality of anchor points and generating anchor point data; storing the anchor point data in the device; determining an initial pose of a virtual object or correcting a display pose of the virtual object based on image data, motion data, and relevant data of the closest anchor point, and updating the display pose of the virtual object based on motion tracking technology. With such a configuration, the display pose of the virtual object is intermittently corrected using the anchor points. On the one hand, the cumulative error is eliminated, and better display accuracy is obtained; on the other hand, the demand for storage and computing resources is less, and it does not rely on a wireless network, reducing the threshold. The augmented reality device is set based on the augmented reality method, which can ensure the precise matching and alignment of the virtual scene and the real scene, and can have a relatively fast operation speed and a relatively high battery life.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and particularly to an augmented reality method and an augmented reality device combined with GNSS data. Background Art

[0002] The core basis of markerless augmented reality technology (AR) is to track the 6DoF (Degree of Freedom) pose of the device in real time (6 degrees of freedom include three-dimensional coordinates and three-dimensional direction angles), and restore the spatial scale. On this basis, virtual elements can be rendered to achieve the fusion of virtual and real. The current process of implementing augmented reality on mobile devices is as follows: collect scene image information through the device camera, collect the linear acceleration and angular acceleration of the device through the IMU (Inertial Measurement Unit), fuse the image information and IMU data to calculate the 6DoF pose of the device and restore the spatial scale, establish a world coordinate system, realize the motion tracking of the device, render virtual elements into the world coordinate system, and align the projection matrix of the camera controlling the rendering of virtual elements with the device camera to achieve the augmented reality function of virtual-real fusion. The process of realizing device pose tracking in augmented reality is also called SLAM (Simultaneous Localization And Mapping), and the accuracy and stability of positioning and tracking are more critical, which determine the effect of augmented reality technology.

[0003] The augmented reality technology itself is not associated with a specific geographical location. When implementing location-based services (LBS), other technical means (such as GPS, Beidou) need to be used for location matching. However, in the civilian consumer field, since the position after GNSS (Global Navigation Satellite System) data calculation can only reach an accuracy of 5-10 meters, it far from meets the accurate alignment and matching requirements of the AR virtual-real environment required by location-based services. Moreover, due to the error accumulation of sensors, after a relatively long time and a relatively large space, SLAM positioning and tracking will drift, resulting in the deviation of the position and direction of virtual objects from real objects.

[0004] In the aspect of precise positioning in large outdoor spaces, the current industry mainly uses sparse, semi-dense, and dense environmental point cloud maps for high-precision positioning of devices in large spaces. Digital modeling of the physical world is completed through the acquisition and calculation of 3D high-precision point cloud maps, and high-precision spatial calculation and positioning of devices are achieved through VPS (Visual Positioning Service) algorithm technology. In theory, centimeter-level positioning and pose determination within 1 degree can be achieved.

[0005] Typical domestic technologies include Huawei's Cyberverse, OPPO's RealVerse, and EasyAR CloudMap of VisionStar Technology, etc.

[0006] For mobile devices with built-in SLAM motion tracking capabilities, the mainstream large-space precise positioning solutions in the industry have the following deficiencies:

[0007] 1. It is necessary to pre-collect and produce high-precision 3D spatial point cloud maps (sparse, semi-dense, dense) of the world environment. The acquisition of high-precision environmental data requires specialized equipment, and the process of constructing high-precision 3D spatial point cloud maps consumes a large amount of computing resources. Usually, it needs to rely on a computer cluster for offline processing and cannot meet the real-time requirements.

[0008] 2. Due to the huge amount of data in high-precision 3D spatial point cloud maps and the computationally intensive positioning and matching process, it is impossible to complete the storage and matching process on mobile terminals. Usually, it is stored in the cloud and the matching calculation is completed in the cloud, which places strong requirements on the mobile device to be connected to the network.

[0009] 3. There is a large amount of information redundancy and useless calculation. For mobile devices with built-in SLAM motion tracking capabilities, the device's own motion tracking capabilities are not utilized at all. When using the VPS algorithm for real-time positioning, there is a large amount of redundant and useless calculation, consuming a large amount of power and computing resources, and significantly reducing the battery life of the device.

[0010] In summary, the existing positioning methods for augmented reality require a large amount of storage space, occupy high computing resources, rely on high wireless network bandwidth and quality, and have problems of a large amount of information redundancy and useless calculation. Summary of the Invention

[0011] The purpose of the present invention is to provide an augmented reality method and augmented reality device that combines GNSS data to solve the problems of the existing positioning methods for augmented reality, which require a large amount of storage space, occupy high computing resources, rely on high wireless network bandwidth and quality, and have problems of a large amount of information redundancy and useless calculation.

[0012] To solve the above technical problems, the present invention provides an augmented reality method combining GNSS data, which is applied to an augmented reality device and includes: setting a plurality of anchor points and generating anchor point data, wherein the anchor point data includes 3D spatial point cloud data and reference position data; storing the anchor point data in the augmented reality device; determining the closest anchor point based on the GNSS position data of the augmented reality device; when meeting a preset condition, constructing comparison point cloud data based on the image data and motion data of the augmented reality device; matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor point, and if the matching is successful, determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the matching result; and when not meeting the preset condition, updating the display pose of the virtual object based on the motion tracking technology of the augmented reality device itself.

[0013] Optionally, the step of setting a plurality of anchor points and generating anchor point data includes: dividing the predicted motion space of the augmented reality device, setting a plurality of the anchor points, wherein the distance between any two adjacent anchor points is within a preset range.

[0014] Optionally, the step of setting a plurality of anchor points and generating anchor point data further includes: for each of the anchor points, performing: setting a virtual reference object and displaying the virtual reference object in a preset pose; obtaining the motion data or a movement instruction, and changing the display pose of the virtual reference object based on the motion data or the movement instruction; obtaining an alignment confirmation instruction; obtaining a first environmental image stream; recording the first motion data when obtaining the first environmental image stream; generating the 3D spatial point cloud data of the anchor point data based on the first environmental image stream and the first motion data; and obtaining at least two GNSS position data of the augmented reality device, performing an average value calculation, and configuring the calculation result as the reference position data.

[0015] Optionally, the step of generating the 3D spatial point cloud data of the anchor point data based on the first environmental image stream and the first motion data includes: extracting feature points of two first image frames in the first environmental image stream based on the ORB algorithm, wherein there is a parallax between the two first image frames; matching the feature points of the two first image frames based on the FLANN algorithm to obtain the corresponding relationship of the feature points of the two first image frames; calculating a first conversion value of the augmented reality device of one of the two first image frames relative to the other based on the method of epipolar geometry, wherein the first conversion value includes: a rotation matrix R and a translation vector t; and summarizing a plurality of the first conversion values, and obtaining the 3D spatial point cloud data with spatial scale information based on triangulation calculation and the pre-integration result of the first motion data.

[0016] Optionally, the step of constructing the comparison point cloud data based on the image data and motion data of the augmented reality device includes: obtaining a second environmental image stream; recording second motion data when obtaining the second environmental image stream; extracting feature points of two second image frames in the second environmental image stream based on the ORB algorithm, where there is a parallax between the two second image frames; matching the feature points of the two second image frames based on the FLANN algorithm to obtain the corresponding relationship of the feature points of the two second image frames; calculating a second transformation value of the augmented reality device of one of the two second image frames relative to the other two frames based on the method of epipolar geometry, where the second transformation value includes: a rotation matrix R and a translation vector t; and, summarizing multiple second transformation values, and obtaining the comparison point cloud data based on triangulation calculation and the pre-integration result of the second motion data.

[0017] Optionally, the step of constructing the comparison point cloud data based on the image data and motion data of the augmented reality device includes: constructing the comparison point cloud data based on an optical flow tracking algorithm.

[0018] Optionally, the step of matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor point and determining the initial pose of the virtual object based on the matching result includes: performing matching calculations based on the RANSAC algorithm and the ICP algorithm to calculate a third transformation value of the comparison point cloud data relative to the 3D spatial point cloud data, or a third transformation value of the 3D spatial point cloud data relative to the comparison point cloud data; and calculating the initial pose of the virtual object based on the third transformation value and the reference position data.

[0019] The step of matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor point and correcting the display pose of the virtual object based on the matching result includes: performing matching calculations based on the RANSAC algorithm and the ICP algorithm to calculate a third transformation value of the comparison point cloud data relative to the 3D spatial point cloud data, or a third transformation value of the 3D spatial point cloud data relative to the comparison point cloud data; calculating a pose transformation matrix based on the third transformation value; and correcting the display pose of the virtual object based on the pose transformation matrix.

[0020] Wherein, the third transformation value includes: a rotation matrix R and a translation vector t.

[0021] Optionally, after determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the matching result, the augmented reality method further includes: correcting the 3D spatial point cloud data of the closest anchor point based on the matching result.

[0022] Optionally, after determining or correcting the current pose information based on the matching result, the augmented reality method further includes: restarting the process of the motion tracking technology or resetting the operating parameters of the motion tracking technology.

[0023] To solve the above technical problems, the present invention also provides an augmented reality device, which includes: a satellite positioning module for obtaining GNSS position data of the augmented reality device; an inertial sensor for obtaining motion data of the augmented reality device; a camera for obtaining image data; a memory for storing anchor data, where the anchor data includes 3D spatial point cloud data and reference position data; a processor for generating anchor data; and further for determining the closest anchor based on the GNSS position data of the augmented reality device; when meeting a preset condition, constructing comparison point cloud data based on the image data and motion data of the augmented reality device; matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor, and determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the matching result; and when not meeting the preset condition, updating the display pose of the virtual object based on the motion tracking technology of the augmented reality device itself; and a display module for displaying the virtual object based on the initial pose or the display pose.

[0024] Compared with the prior art, in the augmented reality method and augmented reality device combining GNSS data provided by the present invention, the augmented reality method includes: setting a plurality of anchors and generating anchor data; storing the anchor data in the device; determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the image data, motion data and relevant data of the closest anchor, and updating the display pose of the virtual object based on the motion tracking technology. With such a configuration, in most cases, the display pose is updated based on the motion tracking technology, and the display pose is intermittently corrected using the anchor. On the one hand, it can eliminate the cumulative error of the motion tracking technology and obtain better positioning accuracy; on the other hand, it requires less storage space and computing resources, and does not rely on a wireless network, reducing the operating threshold of the augmented reality method. The augmented reality device set based on the above augmented reality method can ensure the precise matching and alignment of the virtual scene and the real scene, and can have a faster operation speed and a higher battery life, solving the problems of the positioning method in the prior art that requires a large storage space, occupies high computing resources, depends on a high wireless network bandwidth and quality, and has a large amount of information redundancy and useless calculations. Description of the Drawings

[0025] Those of ordinary skill in the art will understand that the provided drawings are for better understanding of the present invention and do not constitute any limitation to the scope of the present invention. Among them:

[0026] Figure 1 is a schematic flowchart of an augmented reality method according to an embodiment of the present invention;

[0027] Figure 2 is a schematic diagram of anchor point division according to an embodiment of the present invention;

[0028] Figure 3 is a schematic flowchart of generating anchor point data according to an embodiment of the present invention;

[0029] Figure 4 is a schematic flowchart of correcting pose information based on anchor points according to an embodiment of the present invention;

[0030] Figure 5 is a schematic flowchart of virtual scene correction of an augmented reality device according to an embodiment of the present invention. Detailed Embodiments

[0031] To make the objectives, advantages and features of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be noted that the drawings are all in very simplified forms and are not drawn to scale, only for the purpose of facilitating and clearly assisting in explaining the objectives of the embodiments of the present invention. In addition, the structures shown in the drawings are often part of the actual structures. In particular, the emphasis that each drawing needs to show is different, and sometimes different scales are used.

[0032] As used in the present invention, the singular forms "a", "an" and "the" include plural referents, the term "or" is generally used in the sense of including "and / or", the term "several" is generally used in the sense of including "at least one", the term "at least two" is generally used in the sense of including "two or more", in addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first", "second", "third" may explicitly or implicitly include one or at least two of such features. "One end" and "the other end", as well as "proximal end" and "distal end" generally refer to two corresponding parts, which include not only the endpoints. The terms "mounted", "connected", "coupled" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and may be the communication inside two elements or the interaction relationship between two elements. In addition, as used in the present invention, an element being disposed on another element generally only indicates that there is a connection, coupling, cooperation or transmission relationship between the two elements, and the two elements may be directly or indirectly connected, coupled, cooperated or transmitted through an intermediate element, and cannot be construed as indicating or implying the spatial position relationship between the two elements, that is, an element may be inside, outside, above, below or on one side of another element, etc. in any orientation, unless otherwise explicitly specified in the context. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0033] The core idea of the present invention is to provide an augmented reality method and an augmented reality device that combine GNSS data to solve the problems in the prior art that the positioning method for augmented reality requires a large storage space and occupies high computing resources, depends on high wireless network bandwidth and quality, and has a large amount of information redundancy and useless calculations.

[0034] The following is a description with reference to the accompanying drawings.

[0035] Please refer to Figures 1 to 4 , wherein, Figure 1 is a schematic flowchart of the augmented reality method according to an embodiment of the present invention; Figure 2 is a schematic diagram of anchor point division according to an embodiment of the present invention; Figure 3 is a schematic flowchart of generating anchor point data according to an embodiment of the present invention; Figure 4 is a schematic flowchart of correcting pose information based on anchor points according to an embodiment of the present invention; Figure 5 is a schematic flowchart of virtual scene correction of the augmented reality device according to an embodiment of the present invention.

[0036] In LBS-based AR applications, especially when precise matching between the AR scene and the real environment is required, the accuracy after GNSS data calculation cannot provide sufficiently precise positioning and orientation. Due to sensor errors, SLAM motion tracking will inevitably cause cumulative drift, and in some application scenarios, effective correction by loop detection cannot be obtained, resulting in a trend that the position and angle deviation of the AR scene become larger and larger over time and the traveled distance, leading to significant drift of virtual elements in the AR scene after traveling a certain distance, thus causing an obvious deviation between the AR virtual scene and the real physical world, destroying the experience of AR virtual-real fusion, and even making the AR application completely unable to work properly.

[0037] To suppress such drift, the current mainstream approach in the industry is to pre-scan the real scene used by the AR application, generate a high-precision 3D point cloud map and store it in the cloud. When the AR application is launched, by uploading the images captured by the current mobile device camera to the cloud, the cloud uses the VPS algorithm for matching calculation and returns the high-precision pose (position and orientation) of the mobile device in space.

[0038] However, for devices with SLAM motion tracking capabilities themselves, this solution has the aforementioned deficiencies. For some outdoor large-space AR applications, such as river patrol, pipeline patrol, and power line patrol, users will keep walking forward, and there is no possibility of loop for SLAM, so it is impossible to eliminate error cumulative drift by itself. Moreover, these applications cover a large area and have a strong similarity in the real environment. Reconstructing a high-precision 3D point cloud map of the real environment requires a large amount of manpower and resources, the data volume is huge, the calculation amount in the matching process is huge, and the cost is high.

[0039] For the above specific application scenarios, comprehensively using GNSS data positioning, SLAM motion tracking, and AR anchor correction and deviation correction can effectively solve the problem of AR scene cumulative drift. The design idea of the present invention is to transform the overall real environment 3D point cloud reconstruction into one AR anchor after another.

[0040] As Figure 1 shown, based on the above design idea, this embodiment provides an augmented reality method combined with GNSS data, which is applied to an augmented reality device and includes:

[0041] S10 Set multiple anchors and generate anchor data, where the anchor data includes 3D spatial point cloud data and reference position data.

[0042] S20 Store the anchor data in the augmented reality device.

[0043] S30 Based on the GNSS position data of the augmented reality device, determine the closest anchor.

[0044] When S40 meets the preset conditions, comparison point cloud data is constructed based on the image data and motion data of the augmented reality device; the comparison point cloud data is matched with the 3D spatial point cloud data of the closest anchor point. If the match is successful, the initial pose of the virtual object is determined based on the match result or the display pose of the virtual object is corrected.

[0045] In addition, when S50 does not meet the preset conditions, the display pose of the virtual object is updated based on the motion tracking technology of the augmented reality device itself.

[0046] Through the above settings, the augmented reality method has obtained the following beneficial effects:

[0047] 1. It can simplify the data acquisition and 3D spatial point cloud production process, and complete the acquisition, calculation and storage of the point cloud map on the mobile terminal.

[0048] 2. Combining with GNSS positioning data, high-precision positioning of the device is completed locally on the mobile terminal, and the AR scene is matched and aligned with the real space on site.

[0049] 3. By setting a visual correction point (i.e., an anchor point) at a certain distance interval, making full use of the GNSS and the device's own SLAM motion tracking capabilities, the AR scene is corrected and rectified only at specific position points, thus greatly reducing VPS calculations. It not only achieves the ability of high-precision positioning, but also can reduce the spatial matching calculation requirements to the greatest extent.

[0050] The above augmented reality method can make the virtual scene and the real scene of the AR device better aligned and coincident.

[0051] Further, the step of setting multiple anchor points and generating anchor point data includes: dividing the expected motion space of the augmented reality device, and setting multiple anchor points, where the distance between any two adjacent anchor points is within a preset range. In one embodiment, the preset range is 50-100 meters. Adjacent anchor points mean that for one of the anchor points, the other anchor point is the closest one to it. The distance of the anchor points can be calculated based on the reference position data. Such a configuration can reduce the number of VPS calculations. Specifically, the spatial division method can be set arbitrarily. For example, it can be divided by tangent circles or fitting rectangles, or some optimization algorithms can be used for spatial segmentation to ensure that when the device is in the working state, after leaving one anchor point, it can encounter the next anchor point at a better time interval or spatial interval.

[0052] Please refer to Figure 2 , Figure 2 which shows a division result of the anchor points. Figure 2In it, the dashed box represents the predicted movement space of the device, and the solid circles represent the influence ranges of each of the anchor points. When the device (in Figure 2 , it is a mobile device) moves in the Figure 2 shown area, it can continuously encounter new anchor points at a relatively high frequency, thereby continuously correcting its own pose information, achieving a better effect.

[0053] In steps S40 and S50, the preset condition in one embodiment is to judge that the distance from the closest anchor point is less than 5 meters (5 meters can be understood as the preset distance) based on the GNSS position data of the device. Further, the positioning method includes, when the preset condition is not met, displaying a map to show the nearest anchor point, the current position information of the device, and the navigation path from the current position of the device to the nearest anchor point. The above information display is used to guide the user to the nearest anchor point.

[0054] Further, the step of setting multiple anchor points and generating anchor point data further includes: for each of the anchor points, execute:

[0055] S110 Set a virtual reference object and display the virtual reference object in a preset pose.

[0056] S120 Obtain the motion data or movement instruction, and change the display pose of the virtual reference object based on the motion data or movement instruction. During step S120, the user generates the motion data by moving the augmented reality device, and the virtual reference object will change its display position based on the user's movement operation. Or, the augmented reality device will generate an interactive operation interface, and the user can generate the movement instruction based on the operation interface. For example, by dragging the virtual reference object with a finger, the virtual reference object can also change its display position. The above process can also be implemented through a specific alignment program.

[0057] S130 Obtain an alignment confirmation instruction. When the virtual reference object is aligned with the real environment, the user inputs the confirmation instruction based on the operation interface, and the alignment program sends the confirmation instruction through the interface. Or during the process of achieving alignment, an auxiliary program judges whether it is aligned based on the current superimposed image and sends the confirmation instruction at an appropriate time.

[0058] S140 Obtain a first environmental image stream. The image stream includes at least two images, such as videos, etc.

[0059] S150 Record the first motion data when obtaining the first environmental image stream.

[0060] And, based on the first environmental image stream and the first motion data, S160 generates the 3D spatial point cloud data of the anchor data.

[0061] The step of setting a plurality of anchors and generating anchor data further includes: for each of the anchors, after obtaining the alignment confirmation instruction, performing: S170 obtaining at least two GNSS position data of the augmented reality device, calculating an average value, and the calculation result is configured as the reference position data. It should be understood that obtaining the reference position data based on the GNSS position data enables the acquisition work of the anchor data to be realized by a handheld mobile device (such as a mobile phone), thereby reducing the work pressure of the operator. However, in some other embodiments, the reference position data may also be obtained by means of a total station or RTK (Real Time Kinematic), or retrieved from an existing database. The accuracy of the reference position data obtained in these ways is relatively high.

[0062] Specifically, the step of generating the 3D spatial point cloud data of the anchor data based on the first environmental image stream and the first motion data includes:

[0063] S161 extracts the feature points of two first image frames in the first environmental image stream based on the ORB (Oriented FAST and Rotated BRIEF) algorithm, where there is a parallax between the two first image frames.

[0064] S162 matches the feature points of the two first image frames based on the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm to obtain the correspondence relationship of the feature points of the two first image frames.

[0065] S163 calculates, based on the method of epipolar geometry, the first conversion value of the augmented reality device of one of the two first image frames relative to the other, where the first conversion value includes: a rotation matrix R and a translation vector t.

[0066] S164 aggregates a plurality of the first conversion values and obtains the 3D spatial point cloud data with spatial scale information based on triangulation calculation and the pre-integration result of the first motion data.

[0067] In step S161, the two first image frames may be adjacent two frames, for example, the first frame and the second frame, the fifth frame and the sixth frame, and so on. They may also be selected according to other rules, such as the difference between the images obtained under a certain algorithm calculation being within a preset difference range. In step S164, multiple first conversion values are calculated from multiple groups of the two first image frames. For example, the first first conversion value is calculated from the first frame and the second frame, the second first conversion value is calculated from the second frame and the third frame, and the nth first conversion value is calculated from the nth frame and the (n + 1)th frame. Then, the above first conversion values are aggregated and calculated. In an embodiment, as the number of the aggregated first conversion values increases, the number of points in the spatial point cloud first continuously increases and finally converges to a preset number or number range.

[0068] In step S164, there is a corresponding relationship between the first motion data and each frame in the first environmental image stream. By pre-integrating the first motion data and using the corresponding relationship between the two, the specific positions corresponding to the feature points in each frame can be obtained. By aggregating the above data, the complete 3D spatial point cloud data can be obtained.

[0069] The ORB algorithm, FLANN algorithm, method of epipolar geometry, and triangulation calculation in the above steps can be understood according to common knowledge and will not be described in detail here.

[0070] Please refer to Figure 3 and the above process can be understood through Figure 3

[0071] Preferably, the step of constructing the comparison point cloud data based on the image data and motion data of the augmented reality device includes:

[0072] S410 Obtain a second environmental image stream.

[0073] S420 Record the second motion data when obtaining the second environmental image stream.

[0074] S430 Extract the feature points of two second image frames in the second environmental image stream based on the ORB algorithm, where there is a parallax between the two second image frames.

[0075] S440 Match the feature points of the two second image frames based on the FLANN algorithm to obtain the corresponding relationship of the feature points of the two second image frames.

[0076] The S450 calculates a second conversion value of one of the two second image frames relative to the augmented reality device of the other two frames based on the method of epipolar geometry, where the second conversion value includes: a rotation matrix R and a translation vector t.

[0077] And, S460 aggregates multiple second conversion values, and obtains the comparison point cloud data based on triangulation calculation and the pre-integration result of the second motion data.

[0078] The above process is similar to steps S140, S150, and steps S161 - 164, and can be understood with reference to the relevant descriptions of steps S140, S150, and steps S161 - 164.

[0079] It should be understood that the process of constructing the comparison point cloud data can be similar to the process of generating the 3D spatial point cloud data, which can improve the reuse rate of program code. However, in different embodiments, different processes can also be adopted based on different focuses of attention. For example, in one embodiment, the step of constructing the comparison point cloud data based on the image data and motion data of the augmented reality device includes: constructing the comparison point cloud data based on the optical flow tracking algorithm. In other embodiments, other algorithms that can construct the feature point cloud data can also be adopted.

[0080] Preferably, the step of matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor point and determining the initial pose of the virtual object based on the matching result includes: performing matching calculations based on the RANSAC (RANdom SAmpleConsensus) algorithm and the ICP (Iterative Closest Point) algorithm to calculate a third conversion value of the comparison point cloud data relative to the 3D spatial point cloud data, or a third conversion value of the 3D spatial point cloud data relative to the comparison point cloud data; where the third conversion value includes: a rotation matrix R and a translation vector t. And, calculating the initial pose of the virtual object based on the third conversion value and the reference position data. The above process is used to first display the virtual object when the augmented reality device is just powered on.

[0081] The step of matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor point and correcting the display pose of the virtual object based on the matching result includes: performing matching calculations based on the RANSAC algorithm and the ICP algorithm to calculate the third transformation value of the comparison point cloud data relative to the 3D spatial point cloud data, or the third transformation value of the 3D spatial point cloud data relative to the comparison point cloud data; calculating a pose transformation matrix based on the third transformation value; wherein, the third transformation value includes: a rotation matrix R and a translation vector t. And, correcting the display pose of the virtual object based on the pose transformation matrix. The above process is used to display the virtual object after the augmented reality device first displays the virtual object.

[0082] The above process can be understood through Figure 4 understanding.

[0083] The RANSAC algorithm and the ICP algorithm in the above steps can also be understood according to common knowledge, and the pose transformation matrix can be set according to the relevant definitions of specific coordinate axes and direction angles, which will not be described in detail here.

[0084] In an embodiment, after determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the matching result, the augmented reality method further includes: correcting the 3D spatial point cloud data of the closest anchor point based on the matching result. With such a configuration, the data of the original anchor point can be updated to eliminate the influence of environmental changes (such as seasonal changes or external force damage, etc.) on the augmented reality method. Specifically, the correction process can be to select feature points with a higher matching degree as reference points and replace or update feature points with a lower matching degree. For example, both the comparison point cloud data and the 3D spatial point cloud data have 16 points, among which 14 points match each other, and the data of the other 2 points are close but do not meet the matching conditions. At this time, the data in the 3D spatial point cloud data can be replaced with the data in the comparison point cloud data to achieve correction.

[0085] Preferably, after determining or correcting the current pose information based on the matching result, the augmented reality method further includes: restarting the process of the motion tracking technology or resetting the operating parameters of the motion tracking technology. With such a configuration, the cumulative drift influence caused by long-time large-space motion can be further reduced.

[0086] This embodiment also provides an augmented reality device, which includes: a satellite positioning module for obtaining GNSS position data of the augmented reality device; an inertial sensor for obtaining motion data of the augmented reality device; a camera for obtaining image data; a memory for storing anchor data, where the anchor data includes the 3D spatial point cloud data and reference position data; a processor for generating anchor data; and also for determining the closest anchor based on the GNSS position data of the augmented reality device; when meeting preset conditions, constructing comparison point cloud data based on the image data and motion data of the augmented reality device; matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor, and determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the matching result; and when not meeting the preset conditions, updating the display pose of the virtual object based on the motion tracking technology of the augmented reality device itself; and a display module for displaying the virtual object based on the initial pose or the display pose. In this specification, the business use of the augmented reality device is not restricted and can be augmented reality devices in different application fields. For other modules and working principles of the augmented reality device, those skilled in the art can set them according to common knowledge and will not be elaborated here.

[0087] The process of correcting the virtual scene of the augmented reality device is as Figure 5 shown. In one embodiment, the augmented reality device can interact with the point cloud map in the cloud to obtain a more accurate positioning effect. In another embodiment, the augmented reality device can also be a device without network connection, and only perform positioning and virtual scene correction through local point cloud data. Since the augmented reality device corrects the virtual scene based on the positioning method, it also has the beneficial effects of high accuracy, low threshold, fast speed, and long battery life.

[0088] In summary, in an augmented reality method and an augmented reality device combining GNSS data provided by this embodiment, the augmented reality method includes: setting a plurality of anchor points and generating anchor point data; storing the anchor point data in the device; determining an initial pose of a virtual object or correcting a display pose of the virtual object based on image data, motion data, and relevant data of the closest anchor point, and updating the display pose of the virtual object based on a motion tracking technique. With such a configuration, in most cases, the display pose is updated based on the motion tracking technique, and the display pose is intermittently corrected using anchor points. On the one hand, it can eliminate the cumulative error of the motion tracking technique and obtain better positioning accuracy; on the other hand, it requires less storage space and computing resources, and does not rely on a wireless network, reducing the operating threshold of the augmented reality method. The augmented display device set based on the above augmented reality method can ensure the precise matching and alignment of the virtual scene and the real scene, and can have a relatively fast operation speed and a relatively high battery life, solving the problems of the positioning method in the prior art, which requires a large amount of storage space and high computing resources, relies on a relatively high wireless network bandwidth and quality, and has a large amount of information redundancy and useless calculations.

[0089] The above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention based on the above disclosure belong to the protection scope of the technical solution of the present invention.

Claims

1. An augmented reality method combining GNSS data, characterized in that, Applied to an augmented reality device, including: Setting multiple anchor points to generate anchor point data, where the anchor point data includes 3D spatial point cloud data and reference position data; Storing the anchor point data in the augmented reality device; Determining the closest anchor point based on the GNSS position data of the augmented reality device; When meeting a preset condition, constructing comparison point cloud data based on the image data and motion data of the augmented reality device; matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor point, and if the matching is successful, determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the matching result; and, When not meeting the preset condition, updating the display pose of the virtual object based on the motion tracking technology of the augmented reality device itself.

2. The augmented reality method according to claim 1, wherein The step of setting multiple anchor points to generate anchor point data includes: Dividing the predicted motion space of the augmented reality device, and setting multiple anchor points, where the distance between any two adjacent anchor points is within a preset range.

3. The augmented reality method according to claim 2, wherein The step of setting multiple anchor points to generate anchor point data further includes: For each anchor point, execute: Setting a virtual reference object and displaying the virtual reference object in a preset pose; Obtaining the motion data or movement instruction, and changing the display pose of the virtual reference object based on the motion data or movement instruction; Obtaining an alignment confirmation instruction; Obtaining a first environmental image stream; Recording the first motion data when obtaining the first environmental image stream; Based on the first environmental image stream and the first motion data, generating the 3D spatial point cloud data of the anchor point data; and, Obtaining at least two GNSS position data of the augmented reality device, performing an average value calculation, and configuring the calculation result as the reference position data.

4. The augmented reality method according to claim 3, characterized in that The step of generating the 3D spatial point cloud data of the anchor point data based on the first environmental image stream and the first motion data includes: Extracting the feature points of two first image frames in the first environmental image stream based on the ORB algorithm, where there is a parallax between the two first image frames; Matching the feature points of the two first image frames based on the FLANN algorithm to obtain the corresponding relationship of the feature points of the two first image frames; Calculating the first conversion value of the augmented reality device of one of the two first image frames relative to the other based on the method of epipolar geometry, where the first conversion value includes: rotation matrix R and translation vector t; and, Summarizing multiple first conversion values, and obtaining the 3D spatial point cloud data with spatial scale information based on triangulation calculation and the pre-integration result of the first motion data.

5. The augmented reality method according to claim 1, characterized in that The step of constructing comparison point cloud data based on the image data and motion data of the augmented reality device includes: Obtaining a second environmental image stream; Recording the second motion data when obtaining the second environmental image stream; Extracting the feature points of two second image frames in the second environmental image stream based on the ORB algorithm, where there is a parallax between the two second image frames; Match the feature points of the two second image frames based on the FLANN algorithm to obtain the corresponding relationship of the feature points of the two second image frames; Solve the second transformation value of the augmented reality device of one of the two second image frames relative to the other two frames based on the method of epipolar geometry, where the second transformation value includes: a rotation matrix R and a translation vector t; and, Summarize multiple second transformation values, and obtain the comparison point cloud data based on triangulation calculation and the pre-integration result of the second motion data.

6. The augmented reality method according to claim 1, wherein The step of constructing the comparison point cloud data based on the image data and motion data of the augmented reality device includes: constructing the comparison point cloud data based on the optical flow tracking algorithm.

7. The augmented reality method according to claim 5 or 6, characterized in that The step of matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor point and determining the initial pose of the virtual object based on the matching result includes: Perform matching calculations based on the RANSAC algorithm and the ICP algorithm to solve the third transformation value of the comparison point cloud data relative to the 3D spatial point cloud data, or the third transformation value of the 3D spatial point cloud data relative to the comparison point cloud data; and, Calculate the initial pose of the virtual object based on the third transformation value and the reference position data; The step of matching the comparison point cloud data with the 3D spatial point cloud data of the closest anchor point and correcting the display pose of the virtual object based on the matching result includes: Perform matching calculations based on the RANSAC algorithm and the ICP algorithm to solve the third transformation value of the comparison point cloud data relative to the 3D spatial point cloud data, or the third transformation value of the 3D spatial point cloud data relative to the comparison point cloud data; Calculate a pose transformation matrix based on the third transformation value; and, Correct the display pose of the virtual object based on the pose transformation matrix; Wherein, the third transformation value includes: a rotation matrix R and a translation vector t.

8. The augmented reality method according to claim 1, characterized in that After determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the matching result, the augmented reality method further includes: Correct the 3D spatial point cloud data of the closest anchor point based on the matching result.

9. The augmented reality method according to claim 1, wherein After determining the initial pose of the virtual object or correcting the display pose of the virtual object based on the matching result, the augmented reality method further includes: Restart the process of the motion tracking technology or reset the operating parameters of the motion tracking technology.

10. An augmented reality device, characterized in that, The augmented reality device includes: A satellite positioning module for obtaining the GNSS position data of the augmented reality device; An inertial sensor for obtaining the motion data of the augmented reality device; A camera for obtaining image data; A memory for storing anchor point data, where the anchor point data includes 3D spatial point cloud data and reference position data; A processor, configured to generate anchor data; further configured to determine the closest anchor based on the GNSS position data of the augmented reality device; when meeting preset conditions, construct comparison point cloud data based on the image data and motion data of the augmented reality device; match the comparison point cloud data with the 3D spatial point cloud data of the closest anchor, and determine the initial pose of the virtual object or correct the display pose of the virtual object based on the matching result; and when not meeting the preset conditions, update the display pose of the virtual object based on the motion tracking technology of the augmented reality device itself; and, A display module, configured to display the virtual object based on the initial pose or the display pose.

Images