Indoor ar positioning and navigation method and system based on landmark map combined with inertial navigation

Abstract

The application provides an indoor AR positioning and navigation method and system based on a mark graph combined with inertial navigation, relates to the technical field of indoor positioning and navigation, and first deploys a mark graph, uses image recognition to identify the mark graph to obtain positioning information for positioning, and simultaneously combines inertial navigation technology to perform navigation calculation, so that the positioning precision is high; then an A* algorithm is used to plan an optimal path, path data is obtained, a plurality of virtual arrows are generated according to the path data, a virtual path is formed, and then the spatial occlusion relationship between a real environment and the virtual arrows is processed based on a depth map, so that the spatial relationship between the real environment and the virtual arrows is more real, an indoor AR navigation experience of seamless fusion of the real and the virtual is created, and the user is more intuitively and effectively guided to the destination.

Description

Technical Field

[0001] This invention relates to the technical field of indoor positioning and navigation, and more specifically, to an indoor AR positioning and navigation method and system based on a map combined with inertial navigation. Background Technology

[0002] With socio-economic development, skyscrapers are springing up like mushrooms in cities, and people are increasingly relying on navigation services for travel. Traditional navigation services are based on two-dimensional electronic maps, and the navigation information they convey is abstract, requiring time to understand. With the emergence and development of Augmented Reality (AR) technology, navigation services have experienced a leap forward. Unlike traditional navigation services, AR navigation is three-dimensional real-scene navigation. The enhanced real-world images allow users to easily understand navigation information, offering better usability and user experience than flat maps.

[0003] AR navigation systems can be divided into outdoor AR navigation systems and indoor AR navigation systems. Outdoor AR navigation systems are developing more rapidly, and commercial products already exist. However, according to relevant statistics, people spend 80% of their time indoors, and indoor spaces are becoming increasingly large and complex. Therefore, the demand for indoor AR navigation services is growing, but research on indoor AR navigation is currently lagging behind that on outdoor AR navigation.

[0004] The main difference between outdoor and indoor navigation lies in their positioning methods. Outdoor navigation typically uses GPS signals, but indoor GPS signals weaken or even disappear. Furthermore, indoor navigation often requires higher accuracy than GPS, making GPS generally unsuitable for indoor environments. This necessitates alternative positioning methods, which is one of the challenges of indoor navigation. However, many researchers have conducted extensive studies on indoor positioning and AR navigation. For example, they have used Bluetooth beacons to periodically broadcast RSSI signals and employed a constructed RSSI model to calculate the distance between the beacon and the user, updating the location in real time. Simultaneously, virtual landmarks are placed to indicate navigation routes. One existing technology discloses an indoor object positioning system and method that addresses the limitations of mobile devices, such as weak signal reception leading to delayed information updates. This system combines a map generated by a camera and a map generation module with location and region segmentation algorithms. Finally, a classifier network model is used to classify and count different static objects. This also includes a positioning and navigation phase. When the navigation terminal sends a navigation request to the server, based on the received infrared signal and Bluetooth beacon ID, and if the first and second judgment conditions are met, the navigation terminal's location information on the map is determined. After completing path planning, the current positioning status can be fed back to the user. Currently, combining infrared signals and Bluetooth beacons effectively improves positioning accuracy; combining RSSI values ​​and static object classification and statistics based on machine learning algorithms enables reasonable path planning, avoiding resource waste. However, due to the influence of indoor environmental layout, the RSSI model needs to be adjusted according to the surrounding environmental factors of the beacon. In addition, there is also the use of cameras in head-mounted AR visual aids to capture image features in real time as navigation-related landmarks, and to use feature detection matching and location of personal positions. Although high-precision positioning is achieved through AR visual aids, it also raises the barrier to entry, hindering widespread adoption and contradicting the concept of daily travel.

[0005] On the other hand, the spatial relationship between virtual landmarks and the real environment in AR navigation systems is equally important. If virtual landmarks and navigation routes are overlaid on real-world images without obstruction, users will be confused about the navigation route, resulting in a poor user experience. However, how to establish a correct spatial occlusion relationship between virtual objects and real images is a challenge, and current research on AR navigation systems has largely failed to address this issue, including existing commercial outdoor AR navigation systems.

[0006] With the rapid advancements in smart technology and the increasingly frequent upgrades of smartphones, mobile phone performance is constantly improving. Powerful chip processors, professional cameras, and advanced sensors have become the cornerstones for developing augmented reality applications on mobile devices. Meanwhile, mobile devices equipped with depth cameras offer new possibilities for researching highly realistic AR navigation experiences that blend virtual and real environments. Against this backdrop, how to guide users to their destinations more intuitively and effectively, and how to properly handle the spatial relationships between the real environment and virtual objects, has become a pressing issue. Summary of the Invention

[0007] To address the challenges of guiding users to their destinations more intuitively and effectively, and to handle the spatial relationship between the real environment and virtual objects, this invention proposes an indoor AR positioning and navigation method and system based on a landmark map combined with inertial navigation. This method is simple to deploy and low in cost, maintains a correct spatial relationship between the real environment and the virtual path, seamlessly integrates the AR navigation experience, and guides users to their destinations more intuitively and effectively.

[0008] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows:

[0009] An indoor AR positioning and navigation method based on marker maps combined with inertial navigation includes the following steps:

[0010] S1. Construct an indoor map model, and deploy a marker map in the indoor scene corresponding to the indoor map model, using it as an indoor positioning node;

[0011] S2. Collect images and location information of all indoor positioning nodes, and use the AR engine to create an image database from the marker map;

[0012] S3. Use a mobile device to scan the signage in the real environment, and use the AR engine to match the image captured by the mobile device with the image in the image database to obtain the signage name and pose information of the matched image;

[0013] S4. Define an anchor point in the virtual space based on the pose information of the matched image. The position of the anchor point corresponds to the position of the matched image in the real environment. Obtain the image positioning node object in the indoor map model by the identifier name.

[0014] S5. Adjust the position of the indoor map model so that the axis of the image positioning node object is aligned with the anchor point to complete the indoor positioning, and adjust the orientation of the AR camera;

[0015] S6. Calculate the user's current location coordinates in real time using inertial navigation technology;

[0016] S7. Select the target location coordinates, use the A* algorithm to plan the best path based on the navigation grid of the indoor map model, obtain the path data, and generate multiple virtual arrows based on the path data to form a virtual path;

[0017] S8. Based on the depth map, process the spatial occlusion relationship between the real environment and the virtual arrow, and then guide the user to the destination according to the virtual path.

[0018] This technical solution first deploys a signage map, then uses image recognition technology to identify the signage map and obtain positioning information for location. Simultaneously, it combines inertial navigation technology for navigation calculation, achieving high positioning accuracy. Next, it uses the A* algorithm to plan the optimal path, obtains path data, and generates multiple virtual arrows based on this data to form a virtual path. Then, it uses a depth map to process the spatial occlusion relationship between the real environment and the virtual arrows, making the spatial relationship between the real environment and the virtual arrows more realistic. This creates a seamless indoor AR navigation experience that guides users to their destination more intuitively and effectively.

[0019] Preferably, in step S1, an indoor map model is obtained by modeling at a 1:1 scale using modeling software based on the selected floor plan of the indoor scene; when deploying the marker map, suitable locations are selected for deployment in the indoor scene.

[0020] Based on image recognition algorithms, the AR engine is used to match images captured by mobile devices with images in an image database; the pose information includes the position and orientation of the matched image relative to the AR camera in virtual space.

[0021] Here, self-designed images can be used as markers to locate nodes, enriching the diversity of the indoor environment and improving the accuracy of recognition.

[0022] Preferably, the process of adjusting the orientation of the AR camera in step S5 is as follows:

[0023] S1. Make the AR camera face the direction of the anchor point. In the navigation coordinate system, the front of the AR camera is the Z-axis, that is, make the Z-axis face the direction of the anchor point.

[0024] S2. Rotate the AR camera by α degrees around the Y-axis, and then by β degrees around the X-axis to complete the orientation adjustment.

[0025] Preferably, adjusting the orientation of the AR camera involves rotating the AR camera by (β, α, 0) degrees around the XYZ axes of the navigation coordinate system. The value of arctan ranges from (-π / 2, π / 2), and the range of rotation around the axis is 360°. Using atan2 instead of arctan, then:

[0026]

[0027]

[0028] Convert the α and β Euler angles representation to quaternion representation, that is:

[0029] q(α,β,γ)=q y (α)q x (β)q z (γ)

[0030]

[0031]

[0032] Where γ = 0.

[0033] Here, the anchor point corresponds to the position of the matched image in the real environment, that is, the position of the image scanned by the mobile device. This solution converts the α and β Euler angle representation into a quaternion representation to avoid the gimbal deadlock problem that occurs during the application.

[0034] Preferably, before calculating the current position coordinates in real time using inertial navigation technology in step S6, the method further includes converting the carrier coordinate system corresponding to the accelerometer of the user's mobile device into the navigation coordinate system. Let the coordinates of the accelerometer of the mobile device in the carrier coordinate system and the navigation coordinate system be (x, y, z) and (X, Y, Z), respectively. Since the X-axis and Y-axis in the carrier coordinate system and the navigation coordinate system have the same direction, only the Z-axis has a different direction and the two directions are opposite. Therefore, the conversion between the two is completed by a rotation matrix C. The conversion process satisfies the following expression:

[0035]

[0036] Preferably, the process of calculating the current position coordinates in real time using inertial navigation technology is as follows:

[0037] Let the accelerometer reading in the carrier coordinate system be... Then, the acceleration in the vehicle coordinate system is converted to the acceleration in the navigation coordinate system using a rotation matrix. for:

[0038]

[0039] Subtracting the acceleration due to gravity from the acceleration in the navigation coordinate system yields the acceleration a. n :

[0040]

[0041] The user's current location coordinates are obtained based on the user's acceleration.

[0042] Preferably, after obtaining the user's current location coordinates, the marker map is scanned again for repositioning to correct the position. Since the two positioning operations are discontinuous, the position is adjusted using a frame-by-frame correction method to eliminate the jitter phenomenon.

[0043] Preferably, step S7, which generates multiple virtual arrows to form a virtual path, includes:

[0044] S71. Number each coordinate point, and each time take two adjacent coordinate points in the path and calculate the distance between the two adjacent coordinate points;

[0045] S72. Based on the interval of each virtual arrow, calculate the number of virtual arrows required for the virtual path between two coordinate points;

[0046] S73. Each time a virtual path is generated, only the first three coordinate points in the complete path are taken. At the same time, an object pool mode is used to store virtual arrows to avoid frequent creation and destruction of virtual arrows.

[0047] When generating a virtual path, the coordinate point numbers that have already been used are recorded. Coordinate points that have already been used will no longer be used to generate virtual paths. During the user's movement, the best path from the current location to the target location will be updated in real time, and steps S1 to S3 will be repeated until the destination is reached.

[0048] Preferably, the process of processing the spatial occlusion relationship between the real environment and the virtual arrows based on the depth map in step S8 is as follows:

[0049] S81. Use the depth camera of a mobile device to capture images of the real environment in real time, then select keyframe images from the previous images for stereo matching, and obtain two images I after epipolar correction. l and I r , respectively serving as the images from the left and right cameras;

[0050] S82. The depth values ​​of obstructions are calculated using binocular solid geometry, and processed using a planar bilateral solver to obtain high-quality, real-time depth information. The process satisfies:

[0051]

[0052]

[0053]

[0054] This results in a 6-channel image A and a 3-channel image b, where x and y are the coordinates of each pixel, and Z is the depth of the pixel. It is the Hadamard product operator;

[0055] S83. Treat the channels of image A as the upper triangular part of a 3*3 symmetric matrix A', and decompose A' = LDL using LDLT. T Solving the linear system of equations A'x=b, we get the LDL. T x = b, let:

[0056] z = L T x, y = Dz

[0057] Then Ly = b, and we can find the values ​​of x, y, and z:

[0058]

[0059] Use z as the depth value at image coordinates (x, y);

[0060] S84. After obtaining the depth map, virtual and real occlusion is achieved by comparing the depth of the real environment image and the virtual arrow image: if the depth of a certain pixel in the real environment image is greater than the depth of the corresponding pixel in the virtual arrow image, it means that the pixel of the virtual object is in front of the real environment, and the pixel is represented by 1; otherwise, it is represented by 0. Here, 1 represents the pixel that renders the virtual arrow image, and 0 represents the pixel that renders the real environment.

[0061] This application also proposes an indoor AR positioning and navigation system based on a marker map combined with inertial navigation, the system comprising:

[0062] The indoor map building and deployment module is used to build an indoor map model and deploy a marker map in the indoor scene corresponding to the indoor map model, which serves as an indoor positioning node.

[0063] The AR engine module collects images and location information of all indoor positioning nodes and uses the AR engine to create an image database from the signage.

[0064] Mobile devices scan real-world landmarks and use an AR engine to match the images captured by the mobile devices with images in an image database to obtain the landmark name and pose information of the matched images.

[0065] The indoor positioning module defines an anchor point in the virtual space based on the pose information of the matched image. The position of the anchor point corresponds to the position of the matched image in the real environment. The image positioning node object in the indoor map model is obtained by the identifier name.

[0066] The AR camera adjustment module adjusts the position of the indoor map model so that the axis of the image positioning node object is aligned with the anchor point, thus completing indoor positioning, and also adjusts the orientation of the AR camera.

[0067] The inertial navigation module uses inertial navigation technology to calculate the user's current location coordinates in real time; after selecting the target location coordinates, it uses the A* algorithm to plan the optimal path based on the navigation grid of the indoor map model, obtains path data, and generates multiple virtual arrows based on the path data to form a virtual path;

[0068] The system uses depth maps to process the spatial occlusion relationship between the real environment and virtual arrows, and then guides the user to the destination according to the virtual path.

[0069] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0070] This invention proposes an indoor AR positioning and navigation method and system based on marker maps combined with inertial navigation. First, marker maps are deployed, and image recognition technology is used to identify the marker maps and obtain positioning information for location. Simultaneously, inertial navigation technology is combined for navigation calculation, resulting in high positioning accuracy. Then, the A* algorithm is used to plan the optimal path, obtaining path data, and multiple virtual arrows are generated based on the path data to form a virtual path. Finally, the spatial occlusion relationship between the real environment and the virtual arrows is processed based on a depth map, making the spatial relationship between the real environment and the virtual arrows more realistic, creating a seamless indoor AR navigation experience that guides users to their destination more intuitively and effectively. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating the indoor AR positioning and navigation method based on marker maps combined with inertial navigation proposed in Embodiment 1 of the present invention.

[0072] Figure 2 A schematic diagram illustrating the library indoor map model proposed in Embodiment 2 of the present invention;

[0073] Figure 3 A schematic diagram illustrating the library positioning process proposed in Embodiment 2 of the present invention;

[0074] Figure 4 A schematic diagram illustrating the AR camera and its orientation adjustment as proposed in Embodiment 2 of the present invention;

[0075] Figure 5 A schematic diagram illustrating the carrier coordinate system proposed in Embodiment 2 of the present invention;

[0076] Figure 6 A schematic diagram illustrating the navigation coordinate system proposed in Embodiment 2 of the present invention;

[0077] Figure 7 This is a schematic diagram illustrating the imaging principle of the binocular stereo imaging system proposed in Embodiment 2 of the present invention;

[0078] Figure 8A schematic diagram showing the depth of the indoor object proposed in Embodiment 2 of the present invention;

[0079] Figure 9 A schematic diagram illustrating the spatial occlusion processing effect between the real environment and the virtual arrow proposed in Embodiment 2 of the present invention;

[0080] Figure 10 This is a schematic diagram illustrating the overall process of indoor AR positioning and navigation based on an identifier map combined with inertial navigation in Embodiment 2 of the present invention.

[0081] Figure 11 This is a schematic diagram of the indoor AR positioning and navigation system based on marker maps combined with inertial navigation proposed in Embodiment 3 of the present invention;

[0082] Figure 12 This represents the finite state machine diagram proposed in Embodiment 3 of the present invention. Detailed Implementation

[0083] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0084] To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions;

[0085] It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0086] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0087] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0088] Example 1

[0089] In this embodiment, as Figure 1 As shown, an indoor AR positioning and navigation method based on marker maps combined with inertial navigation is proposed, applicable to most indoor scenarios such as hospitals, shopping malls, and airports. This indoor positioning method primarily uses image recognition technology to identify images scanned by mobile devices and matches the correct images in an image database to obtain positioning information. Then, based on this information, the position and orientation of the indoor map model are adjusted to achieve the positioning effect. Finally, inertial navigation technology is used to calculate pedestrian dead reckoning to obtain the user's current location information. The specific steps include:

[0090] S1. Construct an indoor map model, and deploy a marker map in the indoor scene corresponding to the indoor map model, using it as an indoor positioning node;

[0091] S2. Collect images and location information of all indoor positioning nodes, and use the AR engine to create an image database from the marker map;

[0092] S3. Use a mobile device to scan the signage in the real environment, and use the AR engine to match the image captured by the mobile device with the image in the image database to obtain the signage name and pose information of the matched image;

[0093] S4. Define an anchor point in the virtual space based on the pose information of the matched image. The position of the anchor point corresponds to the position of the matched image in the real environment. Obtain the image positioning node object in the indoor map model by the identifier name.

[0094] S5. Adjust the position of the indoor map model so that the axis of the image positioning node object is aligned with the anchor point to complete the indoor positioning, and adjust the orientation of the AR camera;

[0095] S6. Calculate the user's current location coordinates in real time using inertial navigation technology;

[0096] S7. Select the target location coordinates, use the A* algorithm to plan the best path based on the navigation grid of the indoor map model, obtain the path data, and generate multiple virtual arrows based on the path data to form a virtual path;

[0097] S8. Based on the depth map, process the spatial occlusion relationship between the real environment and the virtual arrow, and then guide the user to the destination according to the virtual path.

[0098] Example 2

[0099] In step S1, based on the selected indoor scene floor plan, an indoor map model is obtained by modeling software at a 1:1 scale. In this embodiment, an indoor library is selected, and the resulting indoor map model is as follows. Figure 1 As shown. When deploying the marker map, suitable locations are selected in the indoor scene for deployment, serving as indoor positioning nodes. In actual implementation, self-designed images can be used as marker map positioning nodes to enrich the diversity of the indoor environment and improve recognition accuracy.

[0100] After creating an image database from the logo image using the AR engine, configure this image database in the AR engine's session configuration file.

[0101] Based on image recognition algorithms, the AR engine is used to match images captured by mobile devices with images in an image database; the pose information includes the position and orientation of the matched image relative to the AR camera in virtual space.

[0102] Once all the necessary information is obtained, the indoor map model can be adjusted to achieve the positioning purpose. The process is as follows: Figure 3As shown, an anchor point is defined in the virtual world space using the pose information of the image being matched. The position of this anchor point corresponds to the position of the image being matched in the real environment, that is, the position of the image scanned by the mobile device.

[0103] Then, the image positioning node object in the indoor map model is obtained by the identifier name. This is the virtual object representation of the image scanned by the mobile device in the indoor map. The position of the indoor map model is adjusted so that the pivot of the image positioning node object is aligned with the anchor point. The position of the anchor point corresponds to the position of the matched image in the real environment, that is, the position of the image scanned by the mobile device, thus completing the indoor positioning.

[0104] After the location is updated, the orientation of the AR camera needs to be adjusted so that the front of the AR camera is facing the anchor point. Figure 4 For line segment AB in the diagram, the process of adjusting the orientation of the AR camera described in step S5 is as follows:

[0105] S1. Orient the AR camera so that its front faces the anchor point. In the navigation coordinate system, the front of the AR camera is the Z-axis, meaning the Z-axis should point towards the anchor point. Figure 4 The direction of the median segment AB);

[0106] S2. From Figure 4 As can be seen, the AR camera is rotated α degrees around the Y-axis and then β degrees around the X-axis to complete the orientation adjustment.

[0107] Adjusting the AR camera's orientation involves rotating the AR camera by (β, α, 0) degrees around the XYZ axes of the navigation coordinate system. Since the range of arctan is (-π / 2, π / 2), and the rotation range around the axis is 360°, using atan2 instead of arctan, then:

[0108]

[0109]

[0110] Converting the α and β Euler angles to quaternion representation avoids gimbal deadlock issues during application.

[0111]

[0112] Where γ = 0.

[0113] After the position and orientation are adjusted, the current target position is calculated in real time using inertial navigation technology. The basic principle of inertial navigation is Newton's laws of motion, using the device's built-in gyroscope and accelerometer as sensing elements. When a user moves while holding the mobile device, corresponding acceleration is generated. Integrating this acceleration yields velocity information, and integrating it again yields displacement information. However, the data obtained from the mobile device's accelerometer is relative to the carrier's coordinate system, which is different from Unity3D's three-dimensional spatial coordinate system (hereinafter referred to as the navigation coordinate system). Therefore, it is necessary to convert the carrier's coordinate system to the navigation coordinate system. Figure 5 and Figure 6 As shown, the X and Y axes in the two coordinate systems have the same direction, only the Z axis has a different direction. Before calculating the current position coordinates in real time using inertial navigation technology in step S6, the system further includes converting the carrier coordinate system corresponding to the accelerometer of the user's mobile device into the navigation coordinate system. Let the coordinates of the accelerometer of the mobile device in the carrier coordinate system and the navigation coordinate system be (x, y, z) and (X, Y, Z) respectively. Since the X and Y axes in the carrier coordinate system and the navigation coordinate system have the same direction, only the Z axis has a different direction, and the two directions are opposite, the conversion between the two is completed by the rotation matrix C. The conversion process satisfies the following expression:

[0114]

[0115] Preferably, the process of calculating the current position coordinates in real time using inertial navigation technology is as follows:

[0116] Let the accelerometer reading in the carrier coordinate system be... Then, the acceleration in the vehicle coordinate system is converted to the acceleration in the navigation coordinate system using a rotation matrix. for:

[0117]

[0118] Subtracting the acceleration due to gravity from the acceleration in the navigation coordinate system yields the acceleration a. n :

[0119]

[0120] The user's current location coordinates are obtained based on the user's acceleration.

[0121] After obtaining the user's current location coordinates, the built-in sensors such as the gyroscope and accelerometer in the mobile device inherently have some error. To reduce the impact of sensor errors, an external compensation method is used for position correction. This involves rescanning the marker map for repositioning to correct the location. Since the two positioning operations are discontinuous, a frame-by-frame correction method is used to adjust the position and eliminate the jittery effect. First, the AR marker map is rescanned to obtain a new location, and then this new location is subtracted from the current location to obtain a correction vector α.

[0122] α=(α x ,α y ,α z ) = P0 - P1

[0123] Taking the absolute value of each element in the correction vector α yields α' = (α' x ,α' y ,α' z ), and further subdivide it into multiple components w = (w x ,w y ,w z Assuming 60 frames equal 1 second, the element size within each component is w0 = 0.025. Simultaneously, a direction vector β is generated based on the sign of the elements in the correction vector α.

[0124] β=(β x ,β y ,β z ),in

[0125] Then, in each frame, a component ω is added to the current position P1. The pseudocode of the frame-by-frame correction algorithm is shown below:

[0126]

[0127] After successful location tracking, users can select a target location for navigation. Because virtual arrows need to be generated in real-time to form a virtual path for guiding users to the target location, the number of virtual arrows that need to be created / destroyed is large, which may increase the system load. Therefore, the following steps are performed:

[0128] In the initial stage, after the user selects a target location, the location coordinates of the target location will be obtained;

[0129] After obtaining the location coordinates of the starting point and the ending point, an optimal path is planned using the A* algorithm based on the navigation grid of the indoor map model.

[0130] Once the route is planned, the route data is obtained, and multiple virtual arrows are generated based on the route data to form a virtual path, guiding the user to the destination.

[0131] The process of generating multiple virtual arrows to form a virtual path includes:

[0132] S71. Number each coordinate point, and each time take two adjacent coordinate points in the path and calculate the distance between the two adjacent coordinate points;

[0133] S72. Based on the interval of each virtual arrow, calculate the number of virtual arrows required for the virtual path between two coordinate points;

[0134] S73. Each time a virtual path is generated, only the first three coordinate points in the complete path are taken. At the same time, an object pool mode is used to store virtual arrows to avoid frequent creation and destruction of virtual arrows. All of this is to reduce system overhead.

[0135] When generating a virtual path, the coordinates of previously used points are recorded. These used points will not be used again to generate the virtual path, effectively discarding routes that have already been traversed. As the user moves, the optimal path from the current location to the destination will be updated in real time to prevent the user from deviating from the virtual path's guidance and to promptly replan a new route. Steps S1 through S3 are repeated until the destination is reached.

[0136] Upon reaching the destination, a pop-up window will appear to inform the user that they have arrived. At the same time, the virtual arrows generated during this navigation process will be recycled to the object pool for use in the next navigation session.

[0137] After generating a virtual path based on the path data, the AR camera renders it onto the image. This allows the device to observe a virtual path superimposed on the real scene. However, this is merely an effect created by simply overlaying the virtual path onto the real scene and does not properly handle the spatial occlusion relationship between the two. Therefore, virtual-real occlusion processing is needed to create a more realistic virtual-real fusion effect.

[0138] The process described in step S8, which involves processing the spatial occlusion relationship between the real environment and the virtual arrows based on the depth map, is as follows:

[0139] S81. Use the depth camera of a mobile device to capture images of the real environment in real time, then select keyframe images from the previous images for stereo matching, and obtain two images I after epipolar correction. l and I r , respectively serving as the images from the left and right cameras;

[0140] S82. The depth values ​​of occlusions are calculated using binocular solid geometry, and processed by a planar bilateral solver to obtain high-quality, real-time depth information. Assume that the projection points of spatial point P onto the imaging planes of the left and right cameras are P0 and P1 respectively. l (x l ,yl ) and P r (x r ,y r ),like Figure 7 As shown, the depth value of point P in the point space can be calculated using the following formula.

[0141]

[0142] The depth obtained from the above steps is sparse, temporally inconsistent, and not aligned with the edges of the image. Therefore, a planar bilateral solver is used to process it to obtain high-quality, real-time depth information.

[0143] The process satisfies:

[0144]

[0145]

[0146]

[0147] This results in a 6-channel image A and a 3-channel image b, where x and y are the coordinates of each pixel, and Z is the depth of the pixel. It is the Hadamard product operator;

[0148] S83. Treat the channels of image A as the upper triangular part of a 3*3 symmetric matrix A', and decompose A' = LDL using LDLT. T ,

[0149]

[0150] L and D can be calculated using the following formulas:

[0151]

[0152] Solving the linear system of equations A'x = b, we get the LDL. T x = b, let:

[0153] z = L T x, y = Dz

[0154] Then Ly = b, and we can find the values ​​of x, y, and z:

[0155]

[0156] Use z as the depth value at image coordinates (x, y);

[0157] The depth map is obtained through the above steps, such as... Figure 8As shown, the darker areas in the depth map are farther away from the mobile device. Perform the following steps:

[0158] S84. After obtaining the depth map, occlusion is achieved by comparing the depths of the real environment image and the virtual arrow image: if the depth of a pixel in the real environment image is greater than the depth of the corresponding pixel in the virtual arrow image, it means that the pixel of the virtual object is in front of the real environment, and this pixel is represented by 1; otherwise, it is represented by 0. Here, 1 represents the pixel rendering the virtual arrow image, and 0 represents the pixel rendering the real environment. The final virtual-real fusion effect is shown in the image below. Figure 9 As shown. A schematic diagram of the overall process described above can be found in [reference needed]. Figure 10 .

[0159] Example 3

[0160] like Figure 11 As shown, this embodiment proposes an indoor AR positioning and navigation system based on a marker map combined with inertial navigation. The system includes:

[0161] The indoor map building and deployment module is used to build an indoor map model and deploy a marker map in the indoor scene corresponding to the indoor map model, which serves as an indoor positioning node.

[0162] The AR engine module collects images and location information of all indoor positioning nodes and uses the AR engine to create an image database from the signage.

[0163] Mobile devices scan real-world landmarks and use an AR engine to match the images captured by the mobile devices with images in an image database to obtain the landmark name and pose information of the matched images.

[0164] The indoor positioning module defines an anchor point in the virtual space based on the pose information of the matched image. The position of the anchor point corresponds to the position of the matched image in the real environment. The image positioning node object in the indoor map model is obtained by the identifier name.

[0165] The AR camera adjustment module adjusts the position of the indoor map model so that the axis of the image positioning node object is aligned with the anchor point, thus completing indoor positioning, and also adjusts the orientation of the AR camera.

[0166] The inertial navigation module uses inertial navigation technology to calculate the user's current location coordinates in real time; after selecting the target location coordinates, it uses the A* algorithm to plan the optimal path based on the navigation grid of the indoor map model, obtains path data, and generates multiple virtual arrows based on the path data to form a virtual path;

[0167] The system uses depth maps to process the spatial occlusion relationship between the real environment and virtual arrows, and then guides the user to the destination according to the virtual path.

[0168] In actual implementation, upon entering the system, the inertial navigation module UI is inactive and cannot be used for human-computer interaction. It can only be activated after a successful initial positioning. The mobile device's camera captures a map of markers pre-deployed in the real environment. Then, the image recognition function of the augmented reality engine identifies the image and matches it with the corresponding image in the image database to obtain the image data information. Based on this information, indoor positioning is performed. Once positioning is successful, the inertial navigation module UI is activated.

[0169] The user selects a navigation destination from the UI, and the optimal route is calculated based on the user's location and destination. The program retrieves virtual landmarks from the object pool and sets their position and orientation based on the route data. The device obtains the position and orientation of the AR camera through motion tracking, and the program aligns this with the virtual camera rendering the virtual landmarks. The rendered virtual landmarks are then overlaid on the image captured by the device's camera, making them appear as if they are part of the real environment. To achieve seamless integration of the virtual and real environments, a depth-based occlusion method is used to obtain the distance relationship between the real environment and the virtual objects, thus implementing occlusion between them.

[0170] Users can then use AR navigation based on the rendered virtual road signs.

[0171] The system is divided into three main parts: system management, indoor positioning, and AR navigation.

[0172] The system management section primarily manages the entire lifecycle of the navigation system. It uses a finite state machine to describe the system's state changes throughout the lifecycle and calls corresponding functional modules based on the state. The finite state machine has three states: FINDING, POSITION, and NAVIGATION. The transitions between these three states are as follows: Figure 12 As shown.

[0173] The system initially operates in the FINDING state. In this state, it captures images of pre-AR markers using the mobile device's camera, extracts feature points from these images, and matches these feature points with images in the image database. If a match is successful, the image information is obtained, and the system switches to the POSITION state. If a match fails, the system continuously captures images from each frame for recognition.

[0174] In POSITION state, the system retrieves the object from the indoor map model based on the information of the obtained matching image, performs indoor positioning based on the object's pose information, and switches the state to NAVIGATION if the positioning is successful; otherwise, it switches back to FINDING.

[0175] In NAVIGATION mode, the UI of the inertial navigation module will be activated, and users can operate the relevant UI to select a destination for navigation. The system will respond to the corresponding event handling functions when interacting with the UI.

[0176] The indoor positioning component is mainly used to obtain the user's current location, providing the necessary location information for the AR navigation component.

[0177] The main function of the AR navigation component is to plan an optimal route based on the location coordinates of the starting point and the destination, and to generate a virtual route based on the route data to guide the user to the target location. It also needs to handle the spatial relationship between the real environment and the virtual route.

[0178] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An indoor AR positioning and navigation method based on a map of landmarks combined with inertial navigation, characterized in that, Includes the following steps: S1. Construct an indoor map model, and deploy a marker map in the indoor scene corresponding to the indoor map model, using it as an indoor positioning node; S2. Collect images and location information of all indoor positioning nodes, and use the AR engine to create an image database from the marker map; S3. Use a mobile device to scan the signage in the real environment, and use the AR engine to match the image captured by the mobile device with the image in the image database to obtain the signage name and pose information of the matched image; S4. Define an anchor point in the virtual space based on the pose information of the matched image. The position of the anchor point corresponds to the position of the matched image in the real environment. Obtain the image positioning node object in the indoor map model by the identifier name. S5. Adjust the position of the indoor map model so that the axis of the image positioning node object is aligned with the anchor point to complete the indoor positioning, and adjust the orientation of the AR camera; S6. Calculate the user's current location coordinates in real time using inertial navigation technology; S7. Select the target location coordinates, plan the optimal path using the A* algorithm based on the navigation grid of the indoor map model, obtain path data, and generate multiple virtual arrows to form a virtual path based on the path data; S8. Based on the depth map, process the spatial occlusion relationship between the real environment and the virtual arrow, and then guide the user to the destination according to the virtual path; The process described in step S8, which involves processing the spatial occlusion relationship between the real environment and the virtual arrows based on the depth map, is as follows: S81. Use the depth camera of the mobile device to capture images of the real environment in real time, then select keyframe images from the previous images for stereo matching, and obtain two images after epipolar correction. and , respectively serving as the images from the left and right cameras; S82. The depth values ​​of obstructions are calculated using binocular solid geometry, and processed using a planar bilateral solver to obtain high-quality, real-time depth information. The process satisfies: from which a 6-channel image and a 3-channel image wherein, is the coordinate of each pixel, is the depth of the pixel, is the Hadamard product operator; S83. The image channels are treated as a 3*3 symmetric matrix upper triangular part of which is decomposed by LDLT to solve the linear equations Let:​ Then we have , and we find the value of : Using as image coordinates at depth value; S84. After obtaining the depth map, virtual and real occlusion is achieved by comparing the depth of the real environment image and the virtual arrow image: if the depth of a certain pixel in the real environment image is greater than the depth of the corresponding pixel in the virtual arrow image, it means that the pixel of the virtual object is in front of the real environment, and the pixel is represented by 1; otherwise, it is represented by 0. Here, 1 represents the pixel that renders the virtual arrow image, and 0 represents the pixel that renders the real environment. 2.The indoor AR positioning and navigation method based on the identified map combined with inertial navigation of claim 1, wherein, In step S1, an indoor map model is obtained by modeling at a 1:1 scale using modeling software based on the selected indoor scene floor plan; when deploying the marker map, suitable locations are selected within the indoor scene for deployment. Based on image recognition algorithms, the AR engine is used to match images captured by mobile devices with images in an image database; the pose information includes the position and orientation of the matched image relative to the AR camera in virtual space. 3.The indoor AR positioning and navigation method based on the identified map combined with inertial navigation of claim 1, wherein, The process of adjusting the orientation of the AR camera in step S5 is as follows: S51. Make the AR camera face the direction of the anchor point. In the navigation coordinate system, the front of the AR camera is the Z-axis, that is, make the Z-axis face the direction of the anchor point. S52. Rotate the AR camera about the Y axis by 90 degrees, and then rotate it about the X axis by 90 degrees to complete the orientation adjustment. 4.The indoor AR positioning and navigation method based on the identified map combined with inertial navigation of claim 3, wherein, The process of adjusting the orientation of the AR camera, i.e. the AR camera is rotated around the navigation coordinate system X-Y-Z axes respectively , The value range of , the range of rotation around the axis is 360°, and is used instead of , then: Will and The Euler angle representation is converted to quaternion representation, that is: wherein . 5.The indoor AR positioning and navigation method based on the identified map combined with inertial navigation of claim 1, wherein, Before calculating the current position coordinates in real time using inertial navigation technology in step S6, the method further includes converting the carrier coordinate system corresponding to the accelerometer of the user-held mobile device into the navigation coordinate system. Let the coordinates of the accelerometer of the mobile device in the carrier coordinate system and the navigation coordinate system be respectively... and Since the X and Y axes in the vehicle coordinate system and the navigation coordinate system have the same direction, only the Z axis has a different direction, and the two directions are opposite. Therefore, by using a rotation matrix... Complete the conversion between the two, and the conversion process satisfies the following expression: 。 6.The indoor AR positioning and navigation method based on the identified map combined with inertial navigation of claim 5, wherein, The process of calculating the current position coordinates in real time using inertial navigation technology is as follows: Let the values of the accelerometer in the body coordinate system be Then the acceleration in the navigation coordinate system is obtained by rotating the acceleration in the body coordinate system by the rotation matrix is Subtracting the gravitational acceleration from the acceleration in the navigation coordinate system gives the acceleration : ； The user's current location coordinates are obtained based on the user's acceleration. 7.The indoor AR positioning and navigation method based on the identified map combined with inertial navigation of claim 6, wherein, After obtaining the user's current location coordinates, the marker map is scanned again for repositioning to correct the position. Since the two positioning operations are discontinuous, the position is adjusted using a frame-by-frame correction method to eliminate the jitter. 8.The indoor AR positioning and navigation method based on the identified map combined with inertial navigation of claim 1, wherein, Step S7, which generates multiple virtual arrows to form a virtual path, includes: S71. Number each coordinate point, and each time take two adjacent coordinate points in the path and calculate the distance between the two adjacent coordinate points; S72. Based on the interval of each virtual arrow, calculate the number of virtual arrows required for the virtual path between two coordinate points; S73. Each time a virtual path is generated, only the first three coordinate points of the complete path are taken. At the same time, an object pool mode is used to store virtual arrows to avoid frequent creation and destruction of virtual arrows; When generating a virtual path, the coordinate point numbers that have been used are recorded. Coordinate points that have been used will no longer be used to generate virtual paths. During the user's movement, the best path from the current location to the target location will be updated in real time. Steps S71 to S73 are repeated until the destination is reached.

9. An indoor AR positioning and navigation system based on a map of landmarks combined with inertial navigation, the system being configured to implement the indoor AR positioning and navigation method of claim 1, characterized in that, The system includes: The indoor map building and deployment module is used to build an indoor map model and deploy a marker map in the indoor scene corresponding to the indoor map model, which serves as an indoor positioning node. The AR engine module collects images and location information of all indoor positioning nodes and uses the AR engine to create an image database from the signage. Mobile devices scan real-world landmarks and use an AR engine to match the images captured by the mobile devices with images in an image database to obtain the landmark name and pose information of the matched images. The indoor positioning module defines an anchor point in the virtual space based on the pose information of the matched image. The position of the anchor point corresponds to the position of the matched image in the real environment. The image positioning node object in the indoor map model is obtained by the identifier name. The AR camera adjustment module adjusts the position of the indoor map model so that the axis of the image positioning node object is aligned with the anchor point, thus completing indoor positioning, and also adjusts the orientation of the AR camera. The inertial navigation module uses inertial navigation technology to calculate the user's current location coordinates in real time; after selecting the target location coordinates, it plans the optimal path based on the navigation grid of the indoor map model using the A* algorithm, obtains path data, and generates multiple virtual arrows based on the path data to form a virtual path; The system uses depth maps to process the spatial occlusion relationship between the real environment and virtual arrows, and then guides the user to the destination according to the virtual path.

Images