Intelligent glasses-based multi-source fusion virtual-real registration navigation method

Abstract

The application relates to the technical field of augmented reality navigation, and discloses a multi-source fusion virtual-real registration navigation method based on smart glasses, which comprises the following steps: acquiring multi-source navigation data of the smart glasses in a current navigation period; determining a current registration pose of the smart glasses according to external positioning data, inertial measurement data and visual image data; extracting scene semantic anchor points related to a current navigation path from the visual image data; matching the scene semantic anchor points with map semantic elements in map data to obtain a semantic matching result; correcting a virtual-real mapping relationship corresponding to the smart glasses according to the current registration pose and the semantic matching result; and projecting navigation guide information to a target passing area corresponding to the visual image data according to the corrected virtual-real mapping relationship. The application improves the registration accuracy and display stability of the navigation guide information of the smart glasses and the real passing area.

Description

Technical Field

[0001] This invention relates to the field of augmented reality navigation technology, and more specifically, to a multi-source fusion virtual-real registration navigation method based on smart glasses. Background Technology

[0002] Smart glasses navigation falls at the intersection of augmented reality display, computer vision positioning, multi-source sensor fusion, and interactive path navigation. Smart glasses present navigation routes, turn prompts, directions, and safety reminders from the wearer's first-person perspective through near-eye display devices, allowing users to receive navigation guidance without frequently checking their phone screens while walking, cycling, driving, navigating indoors, or finding their way in complex spaces. With the development of smart glasses, inertial sensors, visual positioning, indoor wireless positioning, and augmented reality display technologies, utilizing smart glasses to combine virtual navigation information with real-world scenarios has become an important technological direction for improving navigation intuitiveness and enhancing traffic safety.

[0003] In existing technologies, such as Chinese invention patent CN116929397B – "A Navigation Method Based on Augmented Reality," the method discloses acquiring the initial and destination location information of a user wearing a head-mounted display device, calculating a navigation path based on the initial and destination location information, displaying the navigation path on the head-mounted display device, acquiring the real-time and predicted location information of the head-mounted display device, and updating the information based on the error between the real-time and predicted location information to improve positioning deviation and navigation delay issues in augmented reality navigation. This method can achieve augmented reality navigation display based on a head-mounted display device and can combine real-time and predicted location information for position error correction.

[0004] However, in first-person perspective navigation scenarios for smart glasses, the accuracy of navigation information display depends not only on whether the wearer's location is updated, but also on whether the virtual navigation guidance information can be stably registered with the real-world area. Outdoor high-rise buildings, tunnels, underground parking garages, indoor shopping malls, airports, and high-speed rail stations can easily cause fluctuations in external positioning data. Weak textures, strong reflections, low light, rapid head turns, dynamic crowd obstructions, or repetitive structures can also easily affect the visual positioning stability corresponding to the first-person perspective image. Even if existing methods can provide error feedback based on real-time and predicted location information, it remains difficult to determine the current registration pose suitable for virtual-real registration display by combining the external positioning reliability state, inertial continuation state, and first-person perspective image feature state within each navigation cycle.

[0005] Meanwhile, existing augmented reality navigation methods typically focus on location updates, path generation, and navigation information display, lacking registration and correction mechanisms for semantic targets in first-person view images. Specifically, existing methods do not fully utilize semantic anchor points in first-person view images, such as road boundaries, lane lines, passage edges, building entrances, floor signs, steps, and elevator entrances, nor do they match these semantic anchor points with map semantic elements in map data to correct the virtual-real mapping relationship between the camera coordinate system, device coordinate system, and map coordinate system. When there are offsets in the positioning results, unstable visual tracking, or rapid changes in head posture, virtual navigation arrows, path lines, or turn prompts are prone to offset from real intersections, passage entrances, or target passage areas, affecting the wearer's judgment of the direction of travel and passage safety.

[0006] Therefore, it is necessary to design a multi-source fusion virtual-real registration navigation method based on smart glasses to solve the problems of existing augmented reality navigation methods, such as difficulty in determining the current registration pose suitable for virtual-real registration based on multi-source navigation data in complex scenarios, lack of virtual-real mapping correction mechanism based on matching first-view scene semantic anchor points with map semantic elements, and easy deviation of navigation guidance information from the actual travel area. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-source fusion virtual-real registration navigation method based on smart glasses, which aims to solve the above-mentioned problems.

[0008] This invention provides a multi-source fusion virtual-real registration navigation method based on smart glasses, comprising: Acquire multi-source navigation data of the smart glasses during the current navigation cycle, wherein the multi-source navigation data includes external positioning data, inertial measurement data, view image data, and map data; The current registration pose of the smart glasses is determined based on the external positioning data, the inertial measurement data, and the viewpoint image data. Extract scene semantic anchor points related to the current navigation path from the perspective image data; The scene semantic anchor points are matched with the map semantic elements in the map data to obtain semantic matching results; Based on the current registration pose and the semantic matching result, the virtual-real mapping relationship corresponding to the smart glasses is corrected. Based on the corrected virtual-real mapping relationship, the navigation guidance information is projected onto the target passage area corresponding to the view image data.

[0009] Furthermore, determining the current registration pose of the smart glasses includes: The reliable result of external positioning is determined based on the signal status of the external positioning data. The inertial continuity result is determined based on the continuous change state of the inertial measurement data; The reliable result of visual positioning is determined based on the image feature state of the aforementioned viewpoint image data; The current registration pose is determined based on the external positioning reliability result, the inertial continuation result, and the visual positioning reliability result.

[0010] Further, determining the current registration pose includes: When the reliable external positioning result meets the external positioning usage conditions, the current registration pose is determined based on the external positioning data and the inertial measurement data; When the external positioning confidence result does not meet the external positioning usage conditions, but the visual positioning confidence result meets the visual positioning usage conditions, the current registration pose is determined based on the viewpoint image data and the inertial measurement data. When neither the external positioning confidence result nor the visual positioning confidence result meets the corresponding usage conditions, the current registration pose is determined based on the registration pose of the previous navigation cycle and the inertial measurement data.

[0011] Furthermore, when extracting the scene semantic anchor points, the process includes: Semantic recognition is performed on the aforementioned viewpoint image data to obtain candidate semantic targets; Based on the predicted projection area of ​​the current navigation path in the view image data, targets that are related to the current navigation path are selected from the candidate semantic targets to obtain the scene semantic anchor point.

[0012] Furthermore, obtaining the semantic matching result includes: Based on the current registration pose, candidate map semantic elements within the field of view of the smart glasses are determined from the map data; The semantic anchor points of the scene are matched with the semantic elements of the candidate map by category and spatial relationship to obtain the semantic matching result.

[0013] Furthermore, when modifying the virtual-real mapping relationship, it includes: The registration offset result of the scene semantic anchor point relative to the corresponding map semantic element is determined based on the semantic matching result; Based on the registration offset results, the mapping relationship between the camera coordinate system, device coordinate system, and map coordinate system is corrected to obtain the corrected virtual-real mapping relationship.

[0014] Furthermore, it also includes: When the registration offset result meets the registration anomaly condition, the navigation anchor point that meets the stability condition in the previous navigation cycle is maintained, and the display level of navigation guidance information in the current navigation cycle is reduced.

[0015] Furthermore, when projecting navigation guidance information onto the target traffic area, this includes: Based on the corrected virtual-real mapping relationship, the current navigation path is converted into the path projection result in the view image data; The target passage area is determined based on the path projection results and the scene semantic anchor points; Navigation guidance information will be anchored and displayed within the target travel area.

[0016] Furthermore, it also includes: Identify abnormal targets ahead of the current navigation path based on the aforementioned view image data; When the abnormal traffic target is located within the target traffic area, a local traffic restriction result is generated; The current navigation path is updated based on the local traffic restriction results, and the navigation guidance information is reprojected based on the updated current navigation path.

[0017] Furthermore, it also includes: The motion state of the wearer is determined based on the inertial measurement data and the viewpoint image data; The display mode of navigation guidance information is determined based on the motion state; When the motion state meets the high-speed movement condition, the display priority of auxiliary navigation information is reduced, while the core navigation guidance information related to the current navigation path is retained; When the motion state meets the stationary condition, the processing frequency of the viewpoint image data and the refresh frequency of the navigation guidance information are reduced.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: By acquiring external positioning data, inertial measurement data, visual image data, and map data of the smart glasses during the current navigation cycle, and combining the reliable results of external positioning, inertial continuity, and reliable results of visual positioning to determine the current registration pose, the smart glasses can still form a pose basis suitable for virtual-real registration display based on multi-source navigation data even when external positioning fluctuates, visual positioning is unstable, or short-term positioning is missing. This reduces the impact of drift from a single positioning source on augmented reality navigation display.

[0019] Simultaneously, this invention extracts scene semantic anchor points related to the current navigation path from the viewpoint image data, and performs category matching and spatial relationship matching between the scene semantic anchor points and map semantic elements in the map data. This allows the virtual-real registration process to no longer rely solely on position coordinates or inertial calculation results, but further incorporates real scene structures within the wearer's current field of vision, such as road boundaries, lane lines, passage edges, building entrances, floor signs, steps, and elevator entrances, as correction criteria. Therefore, it can determine the registration offset result based on the semantic matching results and correct the mapping relationship between the camera coordinate system, device coordinate system, and map coordinate system, improving the stability of the correspondence between navigation guidance information and the actual travel area.

[0020] Furthermore, based on the modified virtual-real mapping relationship, the present invention converts the current navigation path into a path projection result in the view image data, and determines the target passage area by combining scene semantic anchor points. Then, the navigation guidance information is anchored and displayed in the target passage area, so that the turning prompts, path guidance and travel direction prompts can be registered and displayed according to the head posture and field of vision changes of the wearer. This reduces the situation where navigation arrows, path lines or turning prompts deviate from the real intersections, passage entrances or travel areas, and improves the intuitiveness and usability of the first-person view navigation of smart glasses.

[0021] Furthermore, when the registration offset result meets the registration anomaly conditions, this invention maintains the navigation anchor point that meets the stability conditions in the previous navigation cycle and reduces the display level of navigation guidance information in the current navigation cycle. This avoids continuing to output misleading strong guidance information when the positioning or semantic matching is abnormal. When an abnormal target is detected ahead of the current navigation path, a local traffic restriction result can be generated and the current navigation path can be updated, so that the navigation guidance information can be reprojected according to the traffic changes in the real scene. When the wearer is in a high-speed moving state or a stationary state, the display mode and data processing frequency of the navigation guidance information can be adjusted to balance navigation safety, display clarity, and the operating load of the smart glasses. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a multi-source fusion virtual-real registration navigation method based on smart glasses, provided as an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] See Figure 1 As shown, this embodiment of the invention provides a multi-source fusion virtual-real registration navigation method based on smart glasses, including: S100: Acquires multi-source navigation data from the smart glasses during the current navigation cycle. The multi-source navigation data includes external positioning data, inertial measurement data, viewpoint image data, and map data.

[0027] Specifically, the current navigation cycle refers to the processing cycle corresponding to one navigation pose update, virtual-real registration update, and navigation display update performed by the smart glasses. The current navigation cycle can be determined according to the display refresh rate of the smart glasses, the sensor sampling rate, or the update interval set by the navigation application. External positioning data is used to provide the smart glasses with a position reference in global or local space. External positioning data can include one or more of the following: satellite positioning data, indoor wireless positioning data, network-assisted positioning data, and base station ranging positioning data. Inertial measurement data is collected by the inertial measurement unit built into the smart glasses. Inertial measurement data includes at least angular velocity data and acceleration data, used to characterize the attitude changes and displacement trends of the smart glasses within the current navigation cycle. Viewpoint image data is collected by the front-facing camera of the smart glasses. Viewpoint image data is used to characterize the real roads, passages, intersections, building entrances, stairs, elevators, ground boundaries, or other passable structures in the wearer's current field of vision. Map data is used to provide navigation paths and scene spatial benchmarks. Map data can include outdoor electronic maps, indoor maps, 3D maps, road semantic features, building semantic features, access facility semantic features, and current navigation path data.

[0028] When acquiring multi-source navigation data, it is necessary to perform time-correlation processing on external positioning data, inertial measurement data, and viewpoint image data to ensure that data within the same current navigation cycle can be used for the same pose determination and virtual-real registration. If the sampling times of different sensor data are not completely consistent, various types of data can be aligned to the reference time of the current navigation cycle based on the timestamp. Map data is retrieved based on the approximate location of the smart glasses, the current navigation task, and the current field of view, so that semantic map elements that may appear within the current field of view can be subsequently extracted from the map data.

[0029] S200: Determines the current registration pose of the smart glasses based on external positioning data, inertial measurement data, and viewpoint image data.

[0030] Specifically, the current registration pose is not simply position coordinates, but rather the pose result used to register and display virtual navigation information with the real scene. The current registration pose includes at least the position and orientation of the smart glasses in the map coordinate system, and may further include the viewing direction and field of view of the smart glasses' front-facing camera relative to the map coordinate system. External positioning data can provide absolute position reference, but may fluctuate in scenarios with tall buildings, indoor spaces, tunnels, or underground environments. Inertial measurement data can provide continuous attitude changes and short-term displacement trends, but prolonged use alone can easily lead to cumulative errors. Viewpoint image data can provide visual features and scene structure references, but may experience tracking instability in scenarios with weak texture, occlusion, strong reflections, or rapid head turns.

[0031] Therefore, this implementation does not select only a single positioning source as the navigation pose, but uses external positioning data, inertial measurement data, and viewpoint image data together to determine the current registration pose. Specifically, the initial position reference of the smart glasses can be obtained first based on the external positioning data. Then, the attitude change and displacement change within the current navigation cycle can be determined based on the inertial measurement data. Furthermore, the attitude direction and position changes are corrected by combining image feature changes in the viewpoint image data, thereby obtaining a current registration pose suitable for virtual-real registration display. Through this process, the current registration pose can simultaneously reflect the external positioning results, continuous inertial changes, and the actual observation state in the first-viewpoint image, providing a foundation for subsequent semantic anchor matching and navigation guidance information projection.

[0032] S300: Extract scene semantic anchors related to the current navigation path from the view image data.

[0033] Specifically, scene semantic anchors refer to real-world scene targets in first-view image data that can be used to assist in virtual-real registration. Scene semantic anchors are not all the targets identified in the first-view image, but rather targets that have spatial or traversal relationships with the current navigation path. Scene semantic anchors can include one or more of the following: road boundaries, lane lines, pedestrian crossings, intersection edges, passageway edges, building entrances, lobby entrances, floor markings, step edges, elevator entrances, escalator entrances, parking space markings, and indoor signs.

[0034] When extracting scene semantic anchors, target recognition, semantic segmentation, or boundary extraction can be performed on the view image data to obtain candidate semantic targets. Subsequently, these candidate semantic targets are filtered based on the predicted projection area of ​​the current navigation path in the view image data. If a candidate semantic target is located within the predicted projection area of ​​the current navigation path, or near the boundary of the predicted projection area, and its category is related to the direction of travel, turning position, floor change location, or target entrance location, then that candidate semantic target is used as the scene semantic anchor. This method avoids using billboards, pedestrians, vehicle exteriors, decorations, or background targets unrelated to navigation as registration criteria, improving the stability of subsequent semantic matching.

[0035] S400: Match scene semantic anchors with map semantic elements in map data to obtain semantic matching results.

[0036] Specifically, map semantic elements refer to map targets with spatial location and semantic category that are pre-recorded in map data. Map semantic elements can include road boundaries, lane centerlines, intersection nodes, building entrances, passageway boundaries, floor connection points, stair locations, elevator locations, escalator locations, parking areas, indoor store entrances, and directional sign locations. Map semantic elements typically have map coordinates, category labels, spatial orientation, and topological connectivity.

[0037] During matching, candidate map semantic elements within the current visible range are first selected from the map data based on the current registration pose and the field of view parameters of the smart glasses camera. Then, scene semantic anchors in the viewpoint image data are matched with the candidate map semantic elements using category matching and spatial relationship matching. Category matching determines whether the two elements belong to the same or corresponding semantic categories, while spatial relationship matching determines whether their relative positions, directional relationships, arrangement order, and relationship with the current navigation path are consistent. The semantic matching results can include the correspondence between scene semantic anchors and map semantic elements, as well as the matching confidence and registration offset.

[0038] Through this process, virtual-real registration no longer relies solely on external positioning coordinates, but further utilizes the actual visible real scene structure in the first-view image to correspond with semantic elements in the map, thereby providing a basis for subsequent correction of the virtual-real mapping relationship.

[0039] S500: Based on the current registration pose and semantic matching results, correct the virtual-real mapping relationship corresponding to the smart glasses.

[0040] Specifically, the virtual-real mapping relationship refers to the spatial correspondence between navigation paths, map semantic elements, and navigation guidance information in map data and the first-person view display of smart glasses. This virtual-real mapping relationship can include the mapping between the camera coordinate system, device coordinate system, and map coordinate system. The camera coordinate system corresponds to the observation space of the images captured by the front-facing camera of the smart glasses; the device coordinate system corresponds to the posture and motion state of the smart glasses themselves; and the map coordinate system corresponds to the spatial reference of the map data and navigation path.

[0041] After obtaining the current registration pose, a preliminary mapping relationship can be established between the map coordinate system and the device coordinate system, and between the device coordinate system and the camera coordinate system. Subsequently, based on the semantic matching results, it is determined whether there is an offset between the actual position of the scene semantic anchor point in the first-view image and the predicted position of the corresponding map semantic element projected according to the current virtual-real mapping relationship. If an offset exists, the current registration pose or coordinate mapping relationship is corrected based on this offset, so that the projected position of the map semantic element is closer to the real scene semantic anchor point in the first-view image. After correction, the virtual-real mapping relationship can more accurately reflect the spatial correspondence between the real scene currently observed by the smart glasses and the map data.

[0042] S600: Based on the corrected virtual-real mapping relationship, the navigation guidance information is projected onto the target passage area corresponding to the view image data.

[0043] Specifically, navigation guidance information refers to navigation prompts displayed to the wearer through smart glasses. This information can include direction prompts, turn prompts, path lines, stop prompts, floor change prompts, entrance prompts, and safe distance prompts. The target travel area refers to the actual travel area in the viewpoint image data that corresponds to the current navigation path and is suitable for carrying navigation guidance information. The target travel area can be a road travel area, lane area, pedestrian walkway area, indoor corridor area, stairwell entrance area, elevator entrance area, or destination entrance area.

[0044] After determining the corrected mapping relationship between the virtual and real worlds, the current navigation path in the map data is converted into a path projection result in the view image data, and the display position of the navigation guidance information is determined by combining the scene semantic anchor points. If the current navigation path corresponds to a road or passageway direction, the path line or directional arrow is projected to the center of the road, the middle of the passageway, or the inside of the passable boundary. If the current navigation path corresponds to an intersection turn, entrance, or floor change, the turn prompt, entrance prompt, or floor change prompt is anchored near the corresponding scene semantic anchor point. Through this processing, the navigation guidance information can maintain a spatial correspondence with the actual passable area, reducing the possibility of navigation arrows, path lines, or turn prompts being offset relative to the actual road, passageway entrance, or intersection edge.

[0045] Furthermore, determining the current registration pose of the smart glasses includes: determining the reliable external positioning result based on the signal state of the external positioning data; determining the inertial continuation result based on the continuous change state of the inertial measurement data; determining the reliable visual positioning result based on the image feature state of the viewpoint image data; and determining the current registration pose based on the reliable external positioning result, the inertial continuation result, and the reliable visual positioning result.

[0046] Specifically, the external positioning reliability result is used to characterize whether the external positioning data is suitable as a position constraint for the current registration pose within the current navigation cycle. External positioning data can include satellite positioning data, indoor wireless positioning data, or network-assisted positioning data. For satellite positioning data, the signal status can include the positioning solution status, the number of available satellites, the positioning accuracy factor, the positioning data update time, the continuous positioning drift, and the positioning coordinate jump variable. For indoor wireless positioning data, the signal status can include the number of available base stations, the received signal strength, the ranging residual, the spatial distribution of base stations, the positioning data update time, and the continuous positioning drift. If the data update time of the external positioning data meets the requirements of the current navigation cycle, and the positioning solution status, the number of signals, the ranging residual, or the continuous drift meets the preset reliability conditions, then a reliable external positioning result is generated. If the external positioning data exhibits timeouts, jumps, insufficient signal quantity, excessive ranging residual, or abnormal continuous drift, then an unreliable external positioning result is generated.

[0047] The inertial continuity result characterizes whether the inertial measurement data can support short-term continuous estimation of the smart glasses' pose within the current navigation cycle. Inertial measurement data can include angular velocity data, acceleration data, and sampling timestamps. When determining the inertial continuity result based on the continuous changes in the inertial measurement data, it's possible to determine whether there are sampling interruptions, saturated outputs, abrupt noise, or zero-bias anomalies in the angular velocity and acceleration data. Based on the changes in angular velocity, acceleration, and time intervals within continuous navigation cycles, the attitude change trend and short-term displacement change trend of the smart glasses within the current navigation cycle are determined. If the inertial measurement data remains temporally continuous within the current navigation cycle, and the changes in angular velocity and acceleration do not exceed the preset reasonable motion range, a usable inertial continuity result is generated. If the inertial measurement data exhibits time discontinuities, sensor saturation, abrupt changes, or excessive zero-bias drift, a limited inertial continuity result is generated.

[0048] A reliable visual positioning result characterizes whether the viewpoint image data can support visual pose correction within the current navigation cycle. The image feature state of the viewpoint image data can include the number of trackable features, feature distribution range, number of consecutive frame feature matches, optical flow consistency, image sharpness, image occlusion ratio, number of semantic targets, and semantic target stability. If the number of trackable features in the viewpoint image data meets a preset feature quantity condition, the feature distribution covers multiple regions in the viewpoint image data, and the feature matching results or optical flow change results between consecutive frames meet a preset consistency condition, a reliable visual positioning result is generated. If the viewpoint image data has large-area occlusion, significant motion blur, weak texture, strong reflection, dark light, repetitive texture interference, or consecutive frame matching failures, an unreliable visual positioning result is generated.

[0049] Preset reliable conditions, preset reasonable range of motion, and preset consistency conditions can be determined during the smart glasses initialization phase or the navigation application deployment and calibration phase. During calibration, sample navigation data can be collected in open outdoor roads, roads obscured by buildings, underground parking garages, indoor corridors, shopping mall passages, and areas where floors change. The signal state distribution of external positioning data under stable and abnormal navigation states is statistically analyzed. The continuous variation range of inertial measurement data under normal head rotation, rapid head turning, stationary, walking, cycling, or driving states is statistically analyzed. The image feature state distribution of visual image data under normal lighting, low light, strong light, occlusion, weak texture, and dynamic crowd scenes is statistically analyzed. Based on the above sample data, judgment conditions that can distinguish between reliable and unreliable states are determined, thereby avoiding insufficient adaptability to different devices or different scenarios due to the use of fixed empirical values.

[0050] When determining the current registration pose based on the reliable external positioning results, inertial continuation results, and reliable visual positioning results, the corresponding pose determination method can be selected according to different reliable combinations. If the reliable external positioning result is valid, the position provided by the external positioning data is used as the position constraint, and the current registration pose is determined by combining the attitude change amount provided by the inertial measurement data. If the reliable external positioning result is invalid but the reliable visual positioning result is valid, the changes in visual features and scene structure in the viewpoint image data are used as the basis for pose correction, and the current registration pose is determined by combining the inertial measurement data. If neither the reliable external positioning result nor the reliable visual positioning result is valid, but the inertial continuation result is valid, the current registration pose is obtained by using the registration pose of the previous navigation cycle as the basis and combining the inertial measurement data for short-term pose continuation. If none of the reliable external positioning result, the reliable visual positioning result, or the inertial continuation result meets the usage conditions, the registration pose that meets the stability conditions in the previous navigation cycle can be maintained, and the display level of the navigation guidance information can be reduced to avoid outputting navigation guidance information that deviates significantly from the actual travel area.

[0051] In this embodiment, the current registration pose is not simply obtained by directly using a single type of positioning data, but is determined comprehensively based on the reliable state of external positioning data, inertial measurement data, and viewpoint image data. External positioning data provides a global or local position reference, inertial measurement data maintains short-term continuity, and viewpoint image data provides a basis for visual correction within the current field of view. Through this processing, even when external positioning data fluctuates, the first-viewpoint image is temporarily unstable, or the smart glasses undergo rapid attitude changes, a stable pose foundation can still be provided for subsequent scene semantic anchor matching, virtual-real mapping relationship correction, and navigation guidance information projection.

[0052] Furthermore, determining the current registration pose includes: When the reliable external positioning results meet the conditions for using external positioning, the current registration pose is determined based on the external positioning data and inertial measurement data. When the external positioning confidence result does not meet the external positioning usage conditions, but the visual positioning confidence result meets the visual positioning usage conditions, the current registration pose is determined based on the view image data and inertial measurement data. When neither the external positioning confidence result nor the visual positioning confidence result meets the corresponding usage conditions, the current registration pose is determined based on the registration pose and inertial measurement data of the previous navigation cycle.

[0053] Specifically, external positioning data is used to provide a position reference for smart glasses in a map coordinate system. External positioning data can include one or more of the following: satellite positioning data, UWB positioning data, Bluetooth beacon positioning data, and WiFi positioning data. The reliability of external positioning results can be determined based on the data update time, the number of signal sources, positioning accuracy indicators, continuous position hops, and positioning residuals. For example, if the data update time is no more than 1000ms, the number of signal sources is no less than 3, the continuous position hops are no more than 0.5m to 5m, and the positioning residuals are no more than 0.3m to 1.5m, the external positioning results are deemed to meet the usage conditions. If the data update time is greater than 3000ms, or the continuous position hops are greater than 5m to 10m, the external positioning results are deemed not to meet the usage conditions.

[0054] Inertial measurement data is used to characterize the attitude changes and short-term motion changes of smart glasses during the current navigation cycle. Inertial measurement data may include three-axis angular velocity data, three-axis acceleration data, and timestamps. The sampling frequency of the inertial measurement data can be from 50Hz to 400Hz, preferably from 100Hz to 200Hz. When judging the inertial continuity result, it can be based on the time interval between adjacent sampling points, the amplitude of angular velocity change, the amplitude of acceleration change, and data continuity. For example, when the time interval between adjacent inertial sampling points is no greater than 20ms, the absolute value of the angular velocity does not exceed 300° / s to 800° / s, the absolute value of the acceleration is within the range of 0.2g to 3g, and there is no interruption in continuous sampling, the inertial continuity result is determined to meet the usage conditions.

[0055] Viewpoint image data is used to provide visual positioning reference within the current field of view of smart glasses. The reliability of visual positioning results can be determined based on the number of trackable feature points, the number of consecutive frame matches, the image occlusion ratio, the image blur ratio, and optical flow consistency. For example, when the number of trackable feature points in the viewpoint image data is between 80 and 500, the number of successfully matched feature points in consecutive frames is no less than 50, the image occlusion ratio is no greater than 30%, and the image blur ratio is no greater than 20%, the reliable visual positioning result meets the usage conditions. When the number of trackable feature points is less than 30, or the image occlusion ratio is greater than 50%, the reliable visual positioning result does not meet the usage conditions.

[0056] When determining the current registration pose, if the external positioning reliability result meets the usage conditions, the position corresponding to the external positioning data is used as the position constraint, and the attitude change of the smart glasses is determined in conjunction with inertial measurement data to obtain the current registration pose. If the external positioning reliability result does not meet the usage conditions, but the visual positioning reliability result does, the relative pose change is determined based on the continuous frame feature matching results in the viewpoint image data, and the current registration pose is obtained in conjunction with inertial measurement data. If neither the external positioning reliability result nor the visual positioning reliability result meets the usage conditions, the registration pose of the previous navigation cycle is used as the reference, and a short-term continuation is performed in conjunction with inertial measurement data to obtain the current registration pose. The duration of the short-term continuation can be from 0.5s to 3s. If external positioning or visual positioning is not restored after this duration, the display level of navigation guidance information can be reduced.

[0057] For example, a user wearing smart glasses enters a shopping mall from an outdoor road. On the outdoor road, the satellite positioning data update time is 500ms, the number of available satellites is 12, and the continuous position hop is 1.2m. Therefore, the external positioning reliability meets the usage conditions, and the smart glasses determine the current registration pose based on satellite positioning data and inertial measurement data. After the user enters the shopping mall, the satellite positioning data update time increases to 2500ms, the number of available satellites decreases to 3, and the continuous position hop reaches 8m. Therefore, the external positioning reliability does not meet the usage conditions. At this time, the number of trackable feature points in the visual image data is 160, the number of feature points successfully matched in consecutive frames is 110, and the occlusion ratio is 18%. Therefore, the visual positioning reliability meets the usage conditions, and the smart glasses switch to determining the current registration pose based on visual image data and inertial measurement data. If the number of trackable feature points drops to 20 due to pedestrian occlusion within the shopping mall, and the occlusion rate reaches 65%, a short-term continuation will be performed based on the registration pose and inertial measurement data from the previous navigation cycle to avoid sudden changes in navigation guidance information.

[0058] Furthermore, when extracting scene semantic anchor points, the process includes: performing semantic recognition on the view image data to obtain candidate semantic targets; and selecting targets that are related to the current navigation path from the candidate semantic targets based on the predicted projection area of ​​the current navigation path in the view image data to obtain scene semantic anchor points.

[0059] Specifically, the viewpoint image data is collected by the front-facing camera of the smart glasses to reflect the real-time traffic scene within the wearer's current field of vision. When performing semantic recognition on the viewpoint image data, distortion correction, brightness equalization, and scale normalization can be performed first. Then, candidate semantic targets can be identified using a semantic segmentation model, object detection model, or boundary recognition model. Candidate semantic targets can include one or more of the following: road boundaries, lane lines, pedestrian crossings, intersection edges, passageway edges, building entrances, lobby entrances, step edges, elevator entrances, escalator entrances, floor markings, parking space markings, and indoor directional markings.

[0060] During semantic recognition, recognition confidence and size conditions can be set for candidate semantic targets. For example, the recognition confidence of a candidate semantic target can be between 0.50 and 0.95, preferably between 0.70 and 0.90; the minimum area of ​​a candidate semantic target in the viewpoint image data can be between 0.1% and 3% of the total image area; and the number of consecutive frames for a candidate semantic target can be between 2 and 8 frames. Targets that are below the recognition confidence condition, are too small, or only appear briefly in a single frame can be excluded from being considered valid candidate semantic targets to reduce interference from pedestrian occlusion, vehicle passage, reflective areas, or image noise on subsequent registration.

[0061] The predicted projection area of ​​the current navigation path in the view image data is an image region determined based on the current registration pose, the field of view parameters of the smart glasses camera, and the current navigation path in the map data. Specifically, the current navigation path in the map data can be transformed to the viewing direction of the smart glasses camera according to the current registration pose and projected into the view image data to form a path projection line. Then, a projection buffer width is set on both sides of the path projection line to obtain the predicted projection area. The projection buffer width can be 5 to 80 pixels, or 1% to 10% of the image width, preferably 2% to 6% of the image width. In pedestrian navigation scenarios, the projection buffer width can be set larger; in cycling or driving navigation scenarios, the projection buffer width can be further limited by considering lane width or road boundaries.

[0062] When selecting scene semantic anchors from candidate semantic targets, the selection can be based on the positional, categorical, and directional relationships between the candidate semantic target and the predicted projection area. If a candidate semantic target is located within the predicted projection area, or if the overlap between the candidate semantic target and the predicted projection area is 20% to 80%, and the category of the candidate semantic target is consistent with the meaning of the current navigation path, then the candidate semantic target is identified as a scene semantic anchor. If a candidate semantic target is located outside the predicted projection area but is adjacent to it as a traffic boundary, such as a road boundary, passage edge, or lane line, it can also be identified as a scene semantic anchor. If a candidate semantic target is unrelated to the current navigation path, such as a billboard, decoration, distant building facade, temporary pedestrian, or unrelated vehicle, it is not considered a scene semantic anchor.

[0063] For example, a user wearing smart glasses navigates to the elevator on the first floor of a shopping mall. The smart glasses collect visual image data that identifies the left and right edges of the passageway, the store entrance, the elevator entrance, floor signs, and pedestrian targets. Based on the current registration pose, the navigation path from the map data is projected onto the visual image data, predicting that the projection area is located in the middle of the passageway and extends to the elevator entrance. Since the left and right edges of the passageway form a boundary with the predicted projection area, and the elevator entrance is located at the end of the predicted projection area and corresponds to a floor switching node in the navigation path, the left and right edges of the passageway and the elevator entrance are used as scene semantic anchors. The store entrance is located outside the predicted projection area, and the pedestrian target is a dynamically occluded target, therefore it is not used as a scene semantic anchor.

[0064] In this way, scene semantic anchors are not all the targets identified in the view image data, but rather stable targets that have spatial and traffic associations with the current navigation path. This reduces the involvement of irrelevant targets in virtual-real registration and improves the accuracy of subsequent matching between scene semantic anchors and map semantic elements.

[0065] Furthermore, when obtaining semantic matching results, the following are included: Based on the current registration pose, candidate map semantic elements within the field of view of the smart glasses are identified from the map data; the scene semantic anchor points are matched with the candidate map semantic elements by category and spatial relationship to obtain semantic matching results.

[0066] Specifically, the map data pre-records map semantic elements, which may include one or more of the following: road boundaries, lane lines, intersection nodes, pedestrian crossings, passageway edges, building entrances, floor signs, elevator entrances, escalator entrances, stairwell entrances, and parking space signs. Each map semantic element can be configured with an element category, map coordinates, orientation information, and adjacent traffic relationships. After determining the position and viewing direction of the smart glasses in the map coordinate system based on the current registration pose, the map semantic elements within the visible range can be selected from the map data as candidate map semantic elements, based on the field of view of the smart glasses' camera. The camera's horizontal field of view can be 60° to 120°, and its vertical field of view can be 35° to 90°; the visible distance can be set according to the navigation scenario: 3m to 30m for indoor navigation, 10m to 80m for outdoor walking navigation, and 30m to 200m for cycling or driving navigation.

[0067] After identifying candidate map semantic elements, scene semantic anchors are matched with these elements by category. Category matching determines whether the scene semantic anchors identified in the first-view image can form a class or correspondence with the candidate map semantic elements in the map data. For example, "elevator entrance" in the first-view image is preferentially matched with elevator entrance elements in the map data; "passage edge" in the first-view image is preferentially matched with passage boundary elements in the map data; and "lane line" in the first-view image is preferentially matched with lane line elements in the map data. The confidence level of category matching can be set from 0.50 to 1.00, preferably from 0.70 to 0.95; if the confidence level of category matching is lower than 0.50, the correspondence is not considered a valid match.

[0068] After category matching, spatial relationship matching is performed. Spatial relationship matching is used to determine whether the scene semantic anchor point and the candidate map semantic elements are consistent in terms of relative position, arrangement direction, and navigation path relationship. Specifically, it can determine whether the left-right position, up-down position, extension direction, and relative relationship with the path projection area of ​​the scene semantic anchor point in the first-view image match the predicted position of the candidate map semantic elements after projection according to the current registration pose. If the projection position deviation between the scene semantic anchor point and the candidate map semantic elements is no greater than 10 to 80 pixels, or no greater than 1% to 8% of the image width, and the angle between their extension directions is no greater than 5° to 30°, then the spatial relationship matching can be determined to be successful. For close-range scenes such as indoor passages and elevator entrances, the projection position deviation threshold can be set to 10 to 40 pixels; for outdoor roads and distant intersections, the projection position deviation threshold can be set to 30 to 80 pixels.

[0069] For example, a user wearing smart glasses navigates to an elevator in a shopping mall. Based on the current registration pose and camera field of view, the indoor map identifies the left and right boundaries of a passage, the elevator entrance, and two store entrances within the visible range. The left and right edges of the passage and the elevator entrance are extracted from the viewpoint image data as semantic anchor points for the scene. Category matching shows that the left edge of the passage corresponds to the left boundary of the passage in the map, the right edge corresponds to the right boundary of the passage in the map, and the elevator entrance corresponds to the elevator entrance in the map. Further spatial relationship matching shows that the left and right edges of the passage are located on opposite sides of the path projection area, and the elevator entrance is located at the end of the path projection area, consistent with the relative position of the projected semantic elements on the map. Therefore, these three sets of correspondences are used as semantic matching results. If a store entrance is identified in the image, but it is located outside the path projection area and does not correspond to the target node of the current navigation path, it is not considered a valid semantic matching result.

[0070] Semantic matching results can include the correspondence between scene semantic anchors and candidate map semantic elements, category matching results, spatial relationship matching results, and matching confidence. Matching confidence can be determined based on category matching confidence, projection position deviation, direction deviation, and the degree of association with the current navigation path. This approach allows for the selection of map semantic elements from map data that actually correspond to the current field of view of the smart glasses, avoiding the inclusion of invisible or path-irrelevant elements in virtual-real registration correction, thereby improving the reliability of subsequent virtual-real mapping relationship correction.

[0071] Furthermore, when modifying the mapping relationship between virtual and real, the following steps are included: Based on the semantic matching results, determine the registration offset of the scene semantic anchor point relative to the corresponding map semantic elements; based on the registration offset results, correct the mapping relationship between the camera coordinate system, device coordinate system and map coordinate system to obtain the corrected virtual-real mapping relationship.

[0072] Specifically, the camera coordinate system represents the first-view image space captured by the front-facing camera of the smart glasses, the device coordinate system represents the posture and motion state of the smart glasses themselves, and the map coordinate system represents the navigation path and the location of map semantic elements in the map data. The virtual-real mapping relationship is used to transform the navigation path and map semantic elements in the map coordinate system to their display positions in the view image data.

[0073] When determining the registration offset result, the map semantic features can be projected onto the viewpoint image data according to the current registration pose to obtain the predicted projection position. Then, the predicted projection position is compared with the actual recognition position of the corresponding scene semantic anchor point in the viewpoint image data to obtain the registration offset result. The registration offset result can include projection position offset, orientation offset, and scale offset. The projection position offset can be represented by pixel distance, with a value ranging from 10 pixels to 80 pixels; the orientation offset can be represented by the angle between the anchor point boundary direction and the projection direction of the map semantic feature, with a value ranging from 5° to 30°; the scale offset can be represented by the ratio of the difference between the projection size of the map semantic feature and the recognition size of the scene semantic anchor point, with a value ranging from 5% to 40%.

[0074] When correcting the virtual-real mapping relationship based on the registration offset results, priority can be given to scene semantic anchors that meet the stability requirements. Meeting the stability requirements means that the scene semantic anchor appears consecutively for 2 to 8 frames, has a recognition confidence level of 0.70 to 0.95, and its position change within consecutive frames does not exceed 20 to 60 pixels. For example, if the left edge of a channel, the right edge of a channel, and an elevator entrance are identified in the viewpoint image data, and the elevator entrance appears consecutively for 5 frames with a recognition confidence level of 0.86 and a position change of 18 pixels, then the elevator entrance can be used as a stable anchor point for virtual-real mapping correction.

[0075] During the correction process, the position mapping between the map coordinate system and the device coordinate system can be adjusted based on the projection position offset, the attitude mapping between the device coordinate system and the camera coordinate system can be adjusted based on the orientation offset, and the display scale or depth reference of the navigation guidance information in the view image data can be adjusted based on the scale offset. To avoid abrupt changes in navigation guidance information within consecutive navigation cycles, the position correction range within a single navigation cycle can be limited to 0.05m to 1m, and the attitude correction range can be limited to 0.5° to 5°. If the registration offset result exceeds the above correction range, the excess portion can be distributed to subsequent navigation cycles for gradual correction.

[0076] For example, a user wearing smart glasses navigates to an elevator in a shopping mall. Based on the current registration pose, the elevator entrance in the map data is projected to the right-center of the viewpoint image data; the actual elevator entrance, determined by semantic recognition, is located in the center of the viewpoint image data, with a 45-pixel positional offset between the two, and an 8° directional offset for the elevator entrance's border. Since 45 pixels is within the correctable range of 10 to 80 pixels, and 8° is within the correctable range of 5° to 30°, the mapping relationship between the camera coordinate system, device coordinate system, and map coordinate system is corrected based on the aforementioned positional and directional offsets, making the projected elevator entrance position on the map closer to the actual elevator entrance in the first-viewpoint image.

[0077] In this way, the virtual-real mapping relationship is no longer generated directly based solely on the current registration pose. Instead, it is further corrected by utilizing the offset between the scene semantic anchor point and the map semantic elements. This allows subsequent navigation guidance information to be displayed more accurately within the real travel area, reducing the offset of path lines, directional arrows, or entrance prompts relative to the real scene.

[0078] Furthermore, it also includes: When the registration offset result meets the registration anomaly conditions, the navigation anchor point that meets the stability conditions in the previous navigation cycle is maintained, and the display level of navigation guidance information in the current navigation cycle is reduced.

[0079] Specifically, the registration offset result is used to represent the degree of projection deviation between the scene semantic anchor point and the corresponding map semantic feature. Registration anomaly conditions can be determined based on projection position offset, orientation offset, scale offset, and continuous frame offset changes. For example, when the projection position offset is greater than 80 pixels, or greater than 8% of the width of the first-view image; the orientation offset is greater than 30°; the scale offset is greater than 40%; or the registration offset result continuously increases over 3 to 5 consecutive frames, the registration offset result can be determined to meet the registration anomaly conditions. For close-range navigation scenarios such as indoor passageways, elevator entrances, and stairwells, the projection position offset threshold can be set to 40 to 80 pixels; for long-range navigation scenarios such as outdoor roads, intersections, and lane lines, the projection position offset threshold can be set to 60 to 120 pixels.

[0080] A navigation anchor point that meets the stability criteria refers to a scene semantic anchor point that has formed an effective match with map semantic features in the previous navigation cycle and whose position changes smoothly over multiple consecutive navigation cycles. Stability criteria may include: the navigation anchor point appears consecutively for 3 to 10 frames, the recognition confidence level is 0.70 to 0.95, the position change in consecutive frames is no greater than 20 to 60 pixels, and the category matching relationship between the anchor point and the corresponding map semantic features remains unchanged. Meeting these conditions indicates that the navigation anchor point in the previous navigation cycle has good spatial stability and can be used as a temporary display reference when registration anomalies occur in the current navigation cycle.

[0081] Reducing the display level of navigation guidance information means minimizing strong spatial anchoring guidance information and retaining basic guidance information that is less likely to mislead the wearer when registration is abnormal. Specifically, this can involve reducing strongly registered content such as 3D path lines, ground-hugging arrows, and entrance anchor marks to edge direction prompts, text distance prompts, or voice prompts; it can also involve reducing the brightness, transparency, size, or display frequency of navigation guidance information. For example, when the registration offset result meets the registration abnormality conditions, the display of ground-hugging path lines can be stopped, and only basic direction prompts such as "Go straight ahead," "Turn left," and "Enter on the right" can be displayed at the edges of the first-person view image, with the display refresh rate reduced from 30fps to 10fps to 15fps.

[0082] For example, when a user wearing smart glasses navigates to an elevator in a shopping mall, the elevator anchor point has appeared continuously for 6 frames in the previous navigation cycle, with a recognition confidence of 0.88. The positional change in consecutive frames does not exceed 15 pixels, thus the elevator anchor point meets the stability condition. In the current navigation cycle, due to pedestrian occlusion, the edge of the elevator entrance in the first-person view image is partially obscured, and the offset between the projected position of the elevator entrance on the map and the actual recognized position reaches 95 pixels, exceeding the registration anomaly threshold of 80 pixels. At this time, instead of immediately correcting the virtual-real mapping relationship based on the abnormal offset, the elevator anchor point that met the stability condition in the previous navigation cycle is maintained, and the 3D entrance arrow originally anchored at the elevator entrance position is reduced to an edge direction prompt to avoid the navigation arrow suddenly jumping to the wrong position.

[0083] In this way, when semantic recognition is affected by occlusion, changes in lighting, or rapid head turning, the smart glasses will not continue to forcibly display spatial anchoring navigation information according to the abnormal registration result. Instead, they will use the stable navigation anchor point of the previous navigation cycle to maintain short-term continuity and reduce the display level of navigation guidance information, thereby reducing the interference of incorrect registration on the wearer's movement judgment.

[0084] Furthermore, when projecting navigation guidance information onto the target traffic area, this includes: Based on the corrected virtual-real mapping relationship, the current navigation path is converted into a path projection result in the view image data; the target passage area is determined based on the path projection result and scene semantic anchor points; and the navigation guidance information is anchored and displayed within the target passage area.

[0085] Specifically, the corrected virtual-real mapping relationship is used to transform the current navigation path in the map data into the smart glasses' viewpoint image data. The current navigation path can include one or more of the following: a straight path segment, a turning path segment, an entrance path segment, a floor-changing path segment, and a destination arrival path segment. During the transformation, the current navigation path can be mapped into the viewpoint image data according to the corrected mapping relationship between the camera coordinate system, device coordinate system, and map coordinate system to obtain the path projection result. The path projection result can be a path projection line, a path projection strip, or the target point projection position.

[0086] The width of the path projection strip can be determined based on the navigation scenario. In indoor pedestrian navigation scenarios, the path projection strip width can be 3% to 12% of the image width; in outdoor pedestrian navigation scenarios, the path projection strip width can be 2% to 10% of the image width; in cycling or driving navigation scenarios, the path projection strip width can be determined in conjunction with lane lines or road boundaries, preferably not exceeding 60% of the target passage area width. By setting a path projection strip, frequent jumps in the path projection result due to slight head movements of the wearer can be avoided.

[0087] When determining the target passage area, the path projection result can be associated with scene semantic anchor points. If the scene semantic anchor point includes road boundaries, lane lines, or passage edges, the path projection result is restricted to the corresponding boundaries to obtain the road passage area, lane passage area, or indoor passage area. If the scene semantic anchor point includes building entrances, elevator entrances, stairwell entrances, or escalator entrances, the corresponding entrance area and its preset distance range in front are taken as the target passage area. The preset distance range in front can be 0.5m to 5m, preferably 1m to 3m.

[0088] When navigation guidance information is anchored and displayed within the target passage area, the navigation guidance information can include directional arrows, path lines, turning prompts, entrance prompts, floor switching prompts, and arrival prompts. Directional arrows or path lines can be displayed in the center of the target passage area, turning prompts can be displayed at the location where the path projection changes direction, and entrance prompts can be displayed near the scene semantic anchor point corresponding to the building entrance, elevator entrance, or stairwell. To reduce occlusion, the occlusion ratio of the navigation guidance information within the target passage area can be controlled between 5% and 25%, preferably between 10% and 15%; when the occlusion ratio exceeds a preset occlusion threshold, the size, transparency, or display level of the navigation guidance information can be reduced.

[0089] For example, a user wearing smart glasses navigates to an elevator in a shopping mall. Based on the corrected virtual-real mapping relationship, the current navigation path in the map data is projected onto the view image data to obtain a path projection band extending from the middle of the passage to the elevator entrance. The left and right edges of the passage and the elevator entrance have been identified as scene semantic anchor points in the view image data. Therefore, the area between the left and right edges of the passage is determined as the target passage area, and the area approximately 2 meters in front of the elevator entrance is determined as the entrance target area. Subsequently, a straight arrow is anchored in the middle of the passage, and an "Enter Elevator" entrance prompt is anchored near the elevator entrance. If pedestrians obstruct the view, reducing the visible area of ​​the target passage area, the display size of the entrance prompt is reduced, while the directional prompt located at the edge of the image is retained.

[0090] In this way, the navigation guidance information is not simply superimposed on the smart glasses display. Instead, the display position is determined based on the corrected virtual-real mapping relationship, path projection results, and scene semantic anchor points. This allows the navigation guidance information to maintain a correspondence with real roads, passages, entrances, or floor switching areas, thereby improving the accuracy and stability of the augmented reality navigation display on smart glasses.

[0091] Furthermore, it also includes: Identify abnormal targets ahead of the current navigation path based on the view image data; When a target with abnormal passage is located within the target passage area, a local passage restriction result is generated; The current navigation path is updated based on the results of local traffic restrictions, and the navigation guidance information is reprojected based on the updated current navigation path.

[0092] Specifically, abnormal traffic targets refer to targets located ahead of the current navigation path that may affect the wearer's continued passage. Abnormal traffic targets can include one or more of the following: temporary barriers, construction signs, static obstacles, congested crowds, parked vehicles, closed turnstiles, closed doors, temporary closure signs, and hazardous areas on the ground. When identifying abnormal traffic targets, target detection and semantic segmentation can be performed on the viewpoint image data to obtain the abnormal target category, abnormal target location, abnormal target outline, and abnormal target confidence score. The abnormal target confidence score can be between 0.60 and 0.95, preferably between 0.75 and 0.90; the minimum area of ​​the abnormal target in the viewpoint image data can be between 0.2% and 5% of the total image area; the number of consecutive frames can be between 2 and 6 frames to reduce frequent path updates caused by false detections in a single frame.

[0093] To determine whether an abnormal traffic target is located within the target passage area, the image region of the abnormal traffic target can be compared with the target passage area. If the overlap ratio between the abnormal traffic target and the target passage area reaches 20% to 80%, or if the abnormal traffic target horizontally covers 30% to 90% of the width of the target passage area, it can be determined that the abnormal traffic target is within the target passage area. If the abnormal traffic target is located outside the target passage area, or only has a small contact with the edge of the target passage area, no local passage restriction result will be generated.

[0094] When generating localized traffic restriction results, the restricted location, restriction category, and restricted range can be recorded. The restricted location can be transformed into the map coordinate system based on the position of the traffic-prone target in the view image data and the current registration pose; the restricted range can be determined based on the outline of the traffic-prone target, the width of the target's passage area, and a preset safety interval. The preset safety interval can be 0.3m to 2m, preferably 0.5m to 1m for indoor walking scenarios and 1m to 2m for cycling scenarios.

[0095] When updating the current navigation path based on localized traffic restrictions, a localized path update can be performed within a range of 5m to 100m from the current location to the path ahead. In indoor navigation scenarios, the localized path update range can be 5m to 30m; in outdoor walking scenarios, it can be 20m to 80m; and in cycling or driving scenarios, it can be 50m to 200m. When updating the path, priority is given to candidate paths that avoid the restricted area and can reconnect to the original current navigation path. If multiple candidate paths exist, the updated current navigation path can be determined based on detour distance, number of turns, passage width, number of scene semantic anchor points, and motion state adaptation results.

[0096] For example, a user wearing smart glasses is following a navigation path to the elevator entrance in a shopping mall. The visual image data identifies a temporary barrier in the middle of the passageway ahead, with an abnormal target confidence level of 0.86, appearing consecutively for four frames. Furthermore, the overlap between the barrier area and the target passageway is 65%. Therefore, it is determined that the temporary barrier is located within the target passageway, generating a result indicating localized traffic restriction. The smart glasses then search for alternative routes within a 20m radius based on the indoor map, selecting a candidate route that detours through the right-hand passageway and reconnects to the path ahead of the elevator entrance as the updated current navigation path. Subsequently, the path projection result is recalculated based on the updated current navigation path, and the new directional arrows and path lines are projected onto the target passageway corresponding to the right-hand passageway.

[0097] In this way, navigation guidance information can adjust the local path based on the actual abnormal targets appearing in the first-person view image, preventing smart glasses from continuing to anchor the navigation guidance information in the already blocked target passage area, thus improving the practicality and safety of navigation guidance in complex scenarios.

[0098] Furthermore, it also includes: The motion state of the wearer is determined based on inertial measurement data and viewpoint image data; the display mode of navigation guidance information is determined based on the motion state; when the motion state meets the high-speed movement condition, the display priority of auxiliary navigation information is reduced, while the core navigation guidance information related to the current navigation path is retained; when the motion state meets the stationary condition, the processing frequency of viewpoint image data and the refresh frequency of navigation guidance information are reduced.

[0099] Specifically, the motion state can include a stationary state, a low-speed moving state, and a high-speed moving state. The motion state can be determined based on changes in acceleration and angular velocity in inertial measurement data, combined with changes in optical flow across consecutive frames in the viewpoint image data, the displacement of scene semantic anchor points, and image stability. The sampling frequency of the inertial measurement data can be from 50Hz to 400Hz, preferably from 100Hz to 200Hz; the frame rate of the viewpoint image data can be from 15fps to 60fps, preferably from 25fps to 30fps.

[0100] When the acceleration change of the smart glasses is less than 0.05g to 0.15g and the absolute value of the angular velocity is less than 3° / s to 10° / s within a continuous period of 1 to 3 seconds, and the continuous frame displacement of the scene semantic anchor point in the viewpoint image data is less than 3 pixels to 10 pixels, the motion state can be determined to meet the static condition. When the moving speed determined based on inertial measurement data and viewpoint image data is greater than 1.5m / s to 3m / s, or the continuous frame optical flow displacement is greater than 20 pixels to 80 pixels, or the absolute value of the angular velocity is greater than 80° / s to 200° / s, the motion state can be determined to meet the high-speed movement condition. The movement speed threshold can be adjusted according to the navigation mode. In pedestrian navigation, the high-speed movement condition can be set to greater than 1.5m / s to 2.5m / s; in cycling navigation, the high-speed movement condition can be set to greater than 3m / s to 8m / s; and in driving navigation, the high-speed movement condition can be set to greater than 8m / s to 20m / s.

[0101] When determining the display mode of navigation guidance information based on the user's motion status, the information can be divided into core navigation guidance information and auxiliary navigation guidance information. Core navigation guidance information includes the current direction of travel, turn prompts, stop prompts, entrance prompts, and safety prompts; auxiliary navigation guidance information includes remaining distance, estimated arrival time, nearby points of interest, route details, and non-essential text descriptions. Display priority is used to determine the display order, position, area, duration, and hiding order of different navigation guidance information in the first-person view. Display priority is determined based on the degree of relevance of the navigation guidance information to the current navigation path, its relevance to traffic safety, the distance to the next navigation action, and the user's motion status. Information directly related to the current direction of travel, upcoming turn actions, stop actions, obstacle avoidance, and target entrances is designated as high-priority information; information used to supplement remaining distance, estimated arrival time, nearby points of interest, and route details is designated as low-priority information.

[0102] When the motion condition meets the high-speed movement requirement, the display priority of auxiliary navigation information is reduced. Specifically, this includes moving auxiliary navigation information from the target traffic area to the edge of the first-view image data, reducing the display area of ​​auxiliary navigation information, shortening the display duration of auxiliary navigation information, or prioritizing the hiding of auxiliary navigation information when there is a display conflict between core navigation guidance information and auxiliary navigation information. Specifically, this can involve hiding nearby points of interest and route details, reducing the amount of text descriptions, and retaining directional arrows, turn prompts, and safety prompts directly related to the current navigation path. The display area of ​​core navigation guidance information can be controlled to 5% to 15% of the total area of ​​the first-view image, while the display area of ​​auxiliary navigation information can be reduced to 0% to 5% of the total area of ​​the first-view image.

[0103] When the motion state meets the static condition, it indicates that the wearer does not currently require high-frequency updates to the spatial anchoring navigation display. In this case, the processing frequency of the viewpoint image data and the refresh frequency of the navigation guidance information can be reduced. For example, the processing frequency of the viewpoint image data can be reduced from 25fps-30fps to 5fps-15fps, and the refresh frequency of the navigation guidance information can be reduced from 30Hz-60Hz to 5Hz-20Hz. If the static state lasts for more than 10s to 60s, only the destination direction, remaining distance, or next step prompts can be retained to reduce the computing power and power consumption on the smart glasses.

[0104] For example, if a user wearing smart glasses is navigating in an indoor shopping mall, and their acceleration change is less than 0.08g, their angular velocity is less than 5° / s, and the continuous frame displacement at the edge of the passageway and the elevator entrance is less than 5 pixels within 2 seconds, then the motion state meets the static condition. The smart glasses reduce the processing frequency of the viewpoint image data from 30fps to 10fps and the refresh frequency of the navigation guidance information to 10Hz, retaining only the "to the elevator entrance" directional prompt. If the user then rides a bicycle at a high speed, reaching 5m / s, and the continuous frame optical flow displacement reaches 60 pixels, then the motion state meets the high-speed movement condition. The smart glasses hide auxiliary navigation information such as the estimated arrival time and nearby points of interest, retaining only the forward turning arrow and the safe distance prompt.

[0105] In this way, the display mode of navigation guidance information can be adjusted according to the state of motion. This can reduce information obstruction and misleading information when moving at high speed, and reduce the image processing frequency and display refresh rate when stationary, thus balancing navigation safety, display stability and the operating load of the smart glasses.

[0106] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the present invention.

Claims

1. A multi-source fusion virtual-real registration navigation method based on smart glasses, characterized in that, include: Acquire multi-source navigation data of the smart glasses during the current navigation cycle, wherein the multi-source navigation data includes external positioning data, inertial measurement data, view image data, and map data; The current registration pose of the smart glasses is determined based on the external positioning data, the inertial measurement data, and the viewpoint image data. Extract scene semantic anchor points related to the current navigation path from the perspective image data; The scene semantic anchor points are matched with the map semantic elements in the map data to obtain semantic matching results; Based on the current registration pose and the semantic matching result, the virtual-real mapping relationship corresponding to the smart glasses is corrected. Based on the corrected virtual-real mapping relationship, the navigation guidance information is projected onto the target passage area corresponding to the view image data.

2. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 1, characterized in that, Determining the current registration pose of the smart glasses includes: The reliable result of external positioning is determined based on the signal status of the external positioning data. The inertial continuity result is determined based on the continuous change state of the inertial measurement data; A reliable visual positioning result is determined based on the image feature state of the aforementioned viewpoint image data; The current registration pose is determined based on the external positioning reliability result, the inertial continuation result, and the visual positioning reliability result.

3. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 2, characterized in that, Determining the current registration pose includes: When the reliable external positioning result meets the external positioning usage conditions, the current registration pose is determined based on the external positioning data and the inertial measurement data; When the external positioning confidence result does not meet the external positioning usage conditions, but the visual positioning confidence result meets the visual positioning usage conditions, the current registration pose is determined based on the viewpoint image data and the inertial measurement data. When neither the external positioning confidence result nor the visual positioning confidence result meets the corresponding usage conditions, the current registration pose is determined based on the registration pose of the previous navigation cycle and the inertial measurement data.

4. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 1, characterized in that, Extracting the scene semantic anchor points includes: Semantic recognition is performed on the aforementioned viewpoint image data to obtain candidate semantic targets; Based on the predicted projection area of ​​the current navigation path in the view image data, targets that are related to the current navigation path are selected from the candidate semantic targets to obtain the scene semantic anchor point.

5. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 1, characterized in that, Obtaining the semantic matching result includes: Based on the current registration pose, candidate map semantic elements within the field of view of the smart glasses are determined from the map data; The semantic anchor points of the scene are matched with the semantic elements of the candidate map by category and spatial relationship to obtain the semantic matching result.

6. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 5, characterized in that, When correcting the virtual-real mapping relationship, the following is included: The registration offset result of the scene semantic anchor point relative to the corresponding map semantic element is determined based on the semantic matching result; Based on the registration offset results, the mapping relationship between the camera coordinate system, device coordinate system, and map coordinate system is corrected to obtain the corrected virtual-real mapping relationship.

7. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 6, characterized in that, Also includes: When the registration offset result meets the registration anomaly condition, the navigation anchor point that meets the stability condition in the previous navigation cycle is maintained, and the display level of navigation guidance information in the current navigation cycle is reduced.

8. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 1, characterized in that, When projecting navigation guidance information onto the target traffic area, this includes: Based on the corrected virtual-real mapping relationship, the current navigation path is converted into the path projection result in the view image data; The target passage area is determined based on the path projection results and the scene semantic anchor points; Navigation guidance information will be anchored and displayed within the target travel area.

9. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 1, characterized in that, Also includes: Identify abnormal targets ahead of the current navigation path based on the aforementioned view image data; When the abnormal traffic target is located within the target traffic area, a local traffic restriction result is generated; The current navigation path is updated based on the local traffic restriction results, and the navigation guidance information is reprojected based on the updated current navigation path.

10. The multi-source fusion virtual-real registration navigation method based on smart glasses according to claim 1, characterized in that, Also includes: The motion state of the wearer is determined based on the inertial measurement data and the viewpoint image data; The display mode of navigation guidance information is determined based on the motion state; When the motion state meets the high-speed movement condition, the display priority of auxiliary navigation information is reduced, while the core navigation guidance information related to the current navigation path is retained; When the motion state meets the stationary condition, the processing frequency of the viewpoint image data and the refresh frequency of the navigation guidance information are reduced.

Images