AR guide method and system for cultural and travel scene

Abstract

The application discloses an AR guide virtual-real fusion method and system for a travel scene, relates to the technical field of augmented reality, and comprises the following steps: collecting environment image frames in the travel scene, and extracting highlight areas and scene feature points in each image frame; based on the position drift rate and texture stability of the highlight areas in the continuous image frames, jointly determining a dynamic reflection area; performing time sequence tracking on the feature points in the dynamic reflection area, calculating the life cycle parameters thereof, identifying dynamic reflection false feature points according to the position drift rate and the life cycle parameters; reducing the matching weight of the dynamic reflection false feature points in visual SLAM pose estimation, using the remaining feature points for spatial positioning, and spatially anchoring and virtually and realistically fusing and displaying virtual guide content; the application solves the problems of visual SLAM positioning drift and virtual-real fusion mispositioning caused by the misidentification of the dynamic reflection area as a stable feature point in a high-reflection environment.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and more specifically, to an AR-guided virtual-real fusion method and system for cultural and tourism scenarios. Background Technology

[0002] With the development of digital cultural tourism and smart museums, augmented reality (AR) technology is gradually being applied to museums, exhibition halls, and tourist attractions. By overlaying virtual explanation information, 3D models, and historical restoration content onto real scenes, it achieves virtual-real fusion guided tours. Existing AR guided tour systems typically employ visual simultaneous localization and mapping (SLAM) technology, which uses cameras to capture environmental images and utilizes image feature point matching to achieve user spatial positioning and virtual content registration.

[0003] Currently, existing technologies mostly employ visual feature extraction algorithms such as ORB, SIFT, or SURF, combined with inertial sensors and depth sensors to improve positioning stability. However, in highly reflective environments such as museum glass display cases and artifact protective covers, spatial positioning drift is prone to occur. Because glass reflections and mirror reflections from lights create dynamic bright areas, visual SLAM algorithms are easily misidentified as stable feature points.

[0004] When users move while wearing AR devices, the reflective areas shift in position with changing viewing angles, causing unstable feature point matching across consecutive image frames. This leads to a decrease in the number of effective feature points, accumulated pose estimation errors, and spatial anchor point drift. Ultimately, this can result in misalignment between virtual explanation labels, AR navigation information, or 3D reconstruction models and real exhibits, affecting the fusion of virtual and real displays. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present invention provide an AR tour guide virtual-real fusion method and system for cultural and tourism scenarios, which solves the problem of visual SLAM positioning drift and virtual-real fusion misalignment caused by the misidentification of dynamic reflective areas as stable feature points in high reflective environments.

[0006] To achieve the above objectives, the present invention provides the following technical solution: Firstly, this application provides an AR-guided virtual-real fusion method for cultural and tourism scenarios. The method includes: acquiring environmental image frames in cultural and tourism scenarios, extracting highlight areas and scene feature points in each image frame; jointly determining dynamic reflective areas based on the position drift rate and texture stability of highlight areas in consecutive image frames; performing temporal tracking on feature points within the dynamic reflective areas, calculating their lifecycle parameters, and identifying dynamic reflective pseudo-feature points based on the position drift rate and lifecycle parameters; reducing the matching weight of dynamic reflective pseudo-feature points in visual SLAM pose estimation, using the remaining feature points for spatial positioning, and spatially anchoring and displaying virtual-real fusion of virtual tour content.

[0007] In one embodiment, environmental image frames in a cultural and tourism scene are acquired, and highlight areas and scene feature points are extracted from each image frame. This includes: continuously acquiring environmental images of the cultural and tourism scene to generate an environmental image frame sequence; preprocessing each image frame and dynamically determining a brightness threshold based on the brightness distribution of the image frame, and extracting highlight areas with brightness higher than the threshold; performing connected component analysis on the highlight areas to obtain highlight area parameters; extracting scene feature points in non-highlight areas of the image frame, and marking feature points located within or near the edges of the highlight areas as highlight-related feature points; and outputting the highlight area detection results and scene feature point extraction results of the current image frame.

[0008] In one embodiment, the method for obtaining the position drift rate includes: obtaining the detection results of bright areas in consecutive image frames; performing inter-frame region similarity matching based on the parameters of each bright area; establishing the association relationship between bright areas in consecutive frames; calculating the change in center coordinates of the same bright area between adjacent image frames according to the association relationship; and calculating the position drift rate of the bright area in combination with the inter-frame time interval, wherein the position drift rate is used to characterize the degree of position change of the bright area per unit time.

[0009] In one embodiment, jointly determining a dynamic reflective region includes: extracting local texture features of the bright areas in consecutive image frames to generate texture feature vectors; calculating the similarity between the texture feature vectors of the corresponding bright areas in the current frame and the previous frame to obtain a texture stability parameter, which is used to characterize the stability of the texture structure within the bright area; jointly determining the region based on the position drift rate and the texture stability parameter: if the position drift rate is higher than a preset drift rate threshold and the texture stability parameter is lower than a preset texture stability threshold, then it is determined to be a dynamic reflective region; and outputting the position parameter and texture stability parameter of the dynamic reflective region.

[0010] In one embodiment, feature points within the dynamic reflective area are time-series tracked, and their lifecycle parameters are calculated. This includes: filtering feature points located within and near the dynamic reflective area; performing feature descriptor matching between consecutive frames to establish a temporal correspondence between the same feature points; tracking the feature points across consecutive frames and recording their appearance, disappearance, and reappearance states; calculating a feature recovery index based on whether the feature point recovers within a preset time window after disappearing, and the spatial offset and matching confidence during recovery; determining whether the feature point is a short-term occlusion feature point or a dynamically reflective unstable feature point based on the feature recovery index; and counting the number of effective tracking frames for the dynamically reflective unstable feature points in consecutive frames to generate feature point lifecycle parameters.

[0011] In one embodiment, identifying dynamic reflective pseudo-feature points based on the position drift rate and lifecycle parameters includes: acquiring the position drift rate of the bright area, the lifecycle parameters of the feature points, and the matching stability parameters to construct an observation feature set; based on the observation feature set, establishing a probabilistic graphical model of the dynamic reflective state, mapping each observation feature to a local posterior probability, and performing weighted fusion to obtain a comprehensive posterior probability of the dynamic reflective state; determining whether the bright area is a dynamic reflective area based on the comprehensive posterior probability, and marking the feature points located in the area as dynamic reflective pseudo-features.

[0012] In one embodiment, reducing the matching weight of dynamic reflective pseudo-feature points in visual SLAM pose estimation includes: obtaining the initial matching weight and matching confidence of the dynamic reflective pseudo-feature points; generating a weight attenuation coefficient based on the matching confidence, adaptively correcting the initial matching weight to obtain the corrected matching weight; inputting the corrected matching weight into the visual SLAM pose estimation module, and applying weak constraints to the dynamic reflective pseudo-feature points during the reprojection error optimization process to reduce their interference with the pose estimation results.

[0013] In one embodiment, spatial positioning is performed using remaining feature points, and the virtual guide content is spatially anchored and displayed using a virtual-real fusion method. This includes: obtaining a set of remaining feature points after weight adjustment, and establishing a correspondence between two-dimensional image observation points and three-dimensional spatial map points; based on the correspondence and a weighted projection error model, using nonlinear least squares optimization to solve for the camera pose of the current frame; according to the solved camera pose, transforming the three-dimensional anchor points of the virtual guide content from the world coordinate system to the camera coordinate system, and mapping them to the two-dimensional image plane through perspective projection to generate a virtual-real fusion display image; and adjusting the spatial anchor points of the virtual guide content in real time according to the continuous changes in the camera pose to achieve stable cross-frame following display.

[0014] In one embodiment, generating a virtual-real fusion display image includes: acquiring virtual overlay display coordinates and depth image data of the current frame; detecting reflective areas in the current image frame, generating a reflective area probability map, and performing pixel-by-pixel confidence mapping and correction on the depth image accordingly to obtain a confidence depth map; extracting depth values ​​from the confidence depth map based on the virtual overlay display coordinates, constructing the spatial difference between the virtual object and the real scene, and performing graded occlusion determination based on a preset spatial tolerance threshold; continuously adjusting the transparency of the virtual object based on the occlusion determination result, combined with the depth difference and depth confidence weight, generating a rendering layer and fusing it with the current image frame to obtain a virtual-real fusion display image; and performing temporal filtering on the fusion results of consecutive frames to suppress display flicker caused by dynamic changes in reflective areas.

[0015] Secondly, this application provides an AR-guided virtual-real fusion system for cultural and tourism scenarios. This system includes: a highlight area and feature point extraction module, used to acquire environmental image frames in cultural and tourism scenarios and extract highlight areas and scene feature points from each image frame; a dynamic reflective area judgment module, used to jointly determine dynamic reflective areas based on the position drift rate and texture stability of highlight areas in continuous image frames; a pseudo-feature point recognition module, used to perform temporal tracking of feature points within the dynamic reflective area, calculate their lifecycle parameters, and identify dynamic reflective pseudo-feature points based on the position drift rate and lifecycle parameters; and an adaptive SLAM localization and virtual-real fusion module, used to reduce the matching weight of dynamic reflective pseudo-feature points in visual SLAM pose estimation, utilize the remaining feature points for spatial localization, and spatially anchor and display the virtual guide content in a virtual-real fusion manner.

[0016] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: Compared to existing AR navigation and visual SLAM solutions that rely on single features or fixed threshold rules, this invention introduces a joint judgment mechanism of high-brightness area drift rate and texture stability to achieve refined identification of dynamic reflective areas. Furthermore, by combining feature point lifecycle parameters and matching stability analysis, a multi-dimensional collaborative pseudo-feature point identification and suppression system is constructed, effectively eliminating reflective interference features in complex cultural and tourism scenarios such as glass display cases. Simultaneously, a probabilistic graphical model based on Bayesian fusion and an adaptive weight decay mechanism are introduced in the SLAM pose estimation process, enabling the feature contribution to be dynamically adjusted with confidence. Combined with depth confidence decay and continuous occlusion modeling, the transformation of virtual-real fusion from discrete judgment to continuous modeling is realized, ultimately significantly improving the system's positioning stability, anti-interference ability, and spatial consistency and visual continuity of virtual content overlay display in complex lighting and high-reflection environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the AR-guided virtual-real fusion method for cultural and tourism scenarios provided in the embodiments of this application.

[0019] Figure 2 This is a schematic diagram of the structure of an AR-guided virtual-real fusion system for cultural and tourism scenarios provided in an embodiment of this application.

[0020] Figure 3 A comparison chart of lifecycle decay curves provided for embodiments of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Reference Figure 1 As shown in the diagram, the AR-guided virtual-real fusion method for cultural and tourism scenarios provided by this invention includes the following steps: S1, using AR devices to collect environmental image frames in cultural and tourism scenes, and extracting highlighted areas and scene feature points from each image frame, including: The AR device continuously captures environmental images in cultural and tourism scenes using its image acquisition module, and generates a continuous sequence of environmental image frames in chronological order. The environmental images include image data of the display area, the exhibit area, and the scene background area.

[0023] Based on the generated continuous environmental image frame sequence, each image frame is preprocessed, including grayscale processing and brightness normalization processing, in order to reduce the impact of changes in ambient lighting on the image brightness distribution. The brightness normalization process includes adjusting the image brightness distribution based on a histogram equalization method.

[0024] Based on the preprocessed image frames, the pixel brightness distribution in each image frame is statistically analyzed, and pixel regions with brightness values ​​higher than a preset brightness threshold are identified to generate candidate bright regions. The preset brightness threshold is dynamically determined based on the average brightness value and brightness variance of the current image frame, specifically as follows: ; In the formula, To preset the brightness threshold, This represents the average brightness value of the current image frame. The luminance variance of the current image frame. The brightness adjustment coefficient is determined through experimental calibration: multiple historical environmental images of typical cultural and tourism scenarios are acquired, and the overall average brightness value and brightness variance of each image are calculated; different candidate coefficients (e.g., 0.3, 0.5, 0.7...1.5) are used to adjust the brightness of the images, and the grayscale difference or contrast between the normal scene area and the bright reflective area in the adjusted image is statistically analyzed; the coefficient value that can effectively separate the normal scene area and the bright reflective area (e.g., the average grayscale difference between the two types of areas is greater than a set threshold) is selected as the brightness adjustment coefficient. The brightness adjustment coefficient is a real number greater than 0, and in some embodiments, it can be a value between 0.3 and 1.5.

[0025] Connectivity analysis is performed on the candidate highlighted regions, and the corresponding highlighted region parameters are extracted. The highlighted region parameters include the center coordinates, area, and boundary contour parameters of the highlighted region. Based on the extracted highlighted region parameters, feature point detection is performed on the non-highlighted regions in each image frame to extract corner features, edge features, or texture features in the scene to generate a set of scene feature points. The feature point detection employs any one of the following algorithms: ORB feature detection algorithm, FAST corner detection algorithm, or SIFT feature detection algorithm.

[0026] Based on the set of scene feature points, feature descriptors corresponding to each scene feature point are extracted, and the positional correspondence between scene feature points and their respective image regions is established. Based on the established positional correspondence, scene feature points located within or at the edge of the highlighted area are marked to generate a set of highlighted associated feature points. Specifically, when the distance between a scene feature point and the boundary of a highlighted area is less than a preset distance threshold, the scene feature point is determined to be a highlighted associated feature point.

[0027] The preset distance threshold is calibrated in the following way: several image samples containing bright areas are obtained, and the feature point matching success rate at different pixel distances is calculated for feature points near the boundary of the bright area; the pixel distance threshold at which the matching success rate first falls below the preset benchmark value (such as 90%) is determined as the preset distance threshold.

[0028] Based on the generated set of highlighted associated feature points and the set of scene feature points, the results of highlight region detection and scene feature point extraction in the current image frame are output.

[0029] S2 performs correlation analysis on the bright areas in consecutive image frames, calculates the positional drift rate of the bright areas, and combines it with texture stability to jointly determine the dynamic reflective areas.

[0030] In this embodiment, correlation analysis is performed on the highlighted regions in consecutive image frames to calculate the positional drift rate of the highlighted regions, including: Obtain the detection results of the highlighted regions in the current image frame and the previous image frame, and extract the center coordinates, area and boundary contour parameters of each highlighted region; The center coordinates of each highlighted area are two-dimensional pixel coordinates in the image pixel coordinate system.

[0031] Based on the extracted highlight region parameters, region similarity matching is performed between each highlight region in the current image frame and the highlight region in the previous image frame to establish the association relationship between highlight regions in consecutive image frames. The region similarity matching includes center coordinate distance matching, region area similarity matching, and boundary contour overlap matching. The boundary contour overlap matching is calculated using the ratio between the overlapping area of ​​the highlighted region boundary and the area of ​​the region union.

[0032] Based on the established association relationship of the highlighted regions, the change in the center coordinates of the corresponding highlighted regions between consecutive image frames is calculated to generate the position change parameters of the highlighted regions. Based on the generated highlight region position change parameters and the time interval between the current image frame and the previous image frame, calculate the position drift rate of the corresponding highlight region. The position drift rate is used to characterize the degree of position change of the highlighted area per unit time, specifically as follows: ; In the formula, For position drift rate, The coordinates of the center of the highlighted region in the current image frame. These are the center coordinates of the highlighted area in the previous image frame. This is the time interval between the current image frame and the previous image frame.

[0033] Furthermore, dynamic reflective areas are jointly determined by combining texture stability, including: Based on the acquired highlighted areas, local texture features are extracted within each highlighted area to generate the texture feature vector of the corresponding highlighted area. The local texture features include at least one of grayscale gradient features, edge direction features, local binary texture features, or texture direction histogram features.

[0034] Based on the generated texture feature vector, the texture similarity is calculated for the corresponding highlighted areas in the current image frame and the previous image frame to generate texture stability parameters. The texture stability parameter is used to characterize the stability of the texture structure within the highlight area across consecutive image frames, specifically: ; In the formula, For texture stability parameters, This is the texture feature vector corresponding to the highlighted area in the current image frame. This is the texture feature vector of the corresponding highlighted area in the previous image frame.

[0035] Based on the generated texture stability parameters, a stability analysis is performed on the texture changes within the highlighted area. Specifically, when the texture stability parameter is lower than the preset texture stability threshold, it is determined that there is a change in texture structure in the corresponding highlighted area.

[0036] Based on the analysis results of positional drift rate and texture stability, the highlighted areas are jointly determined; Specifically, when the position drift rate of the highlighted area is higher than the preset drift rate threshold and the texture stability parameter is lower than the preset texture stability threshold, the corresponding highlighted area is determined to be a dynamic reflective area. When the position drift rate of the highlighted area is lower than the preset drift rate threshold and the texture stability parameter is higher than the preset texture stability threshold, the corresponding highlighted area is determined to be a real highlighted texture area.

[0037] The preset drift rate threshold and preset texture stability threshold are set using empirical parameters. By collecting historical cultural and tourism scene image samples and calculating the position drift rate and texture stability parameters of their bright areas, statistical analysis is performed based on the classification results of manually labeled real reflective areas and real bright texture areas, and the corresponding values ​​that can make the two types of areas distinguish the lowest are selected as the preset drift rate threshold and preset texture stability threshold.

[0038] Table 1 shows the comparison of the false recognition rates of the method of the present invention and the traditional single brightness threshold method in the identification of reflective areas.

[0039]

[0040] This table illustrates that by introducing texture stability constraints, the present invention significantly reduces the misclassification rate in complex reflective scenes, thereby improving the stability of SLAM features.

[0041] Based on the determined dynamic reflective regions, dynamic reflective region marking results are generated, and the position parameters and texture stability parameters of the corresponding dynamic reflective regions are output.

[0042] It should be noted that by jointly determining the positional drift rate of the highlighted area and the texture stability parameter, the highlighted area can be comprehensively analyzed from two independent dimensions: "spatial motion consistency" and "local texture structure stability". Compared with the existing conventional methods that only determine based on positional or brightness changes, this method can effectively distinguish normal highlighted area changes caused by the movement of real objects or changes in real highlighted textures, thereby improving the accuracy of visual SLAM positioning and the continuity and robustness of AR virtual-real fusion display. By introducing a joint constraint mechanism between texture temporal stability and spatial drift characteristics, the recognition of reflective areas is expanded from a single geometric feature judgment to a multi-feature collaborative judgment, improving the adaptability to complex lighting and material environments in cultural and tourism industries.

[0043] S3, perform continuous frame-to-frame temporal tracking on feature points located in the dynamic reflective area, and calculate the lifetime parameters of the corresponding feature points, including: The dynamic reflective area marking results in the current image frame are obtained, and feature points located within the dynamic reflective area and its preset neighborhood are filtered to generate a set of feature points to be tracked. Based on the set of feature points to be tracked, continuous frame feature descriptor matching is performed on each feature point, and the correspondence between feature points in adjacent image frames is established. Among them, continuous frame feature descriptor matching for each feature point refers to calculating the descriptor similarity of feature points in adjacent image frames. The similarity is calculated using Hamming distance or Euclidean distance or a combination thereof. The feature points with the most similar appearance features in different frames are paired up, thereby establishing the correspondence of the same spatial point in the time series.

[0044] Based on the established feature point correspondence, continuous frame temporal tracking is performed on each feature point, and its feature point state information in each image frame is recorded, including appearance state, disappearance state and reappearance state. The reappearance state is the state in which the feature point is successfully matched again within the spatial neighborhood within a preset time window after it disappears.

[0045] Based on the recorded feature point status information, the time node of the first disappearance of the feature point is marked, and the recovery of the feature point in the spatial neighborhood is continuously detected within a preset time window to generate feature recovery detection results; Based on the feature recovery detection results, the feature recovery index is calculated by combining the spatial offset between the reappearance location of the feature point and the previous valid location and the matching confidence. The specific formula for calculating the feature recovery index is as follows: ; In the formula, The feature recovery index is a quantitative indicator used to characterize the reliability of feature recovery when a feature point reappears in its spatial neighborhood after disappearing, considering the combined effects of its spatial location shift and the confidence level of rematching. This refers to the spatial offset of the reappearance location of the feature point relative to its original location. The confidence level of the feature points is rematched by quantifying the descriptor similarity between the re-emerging feature points and the reference feature points. For example, the matching probability value between 0 and 1 is obtained by normalizing the Hamming distance or Euclidean distance.

[0046] The spatial offset is the Euclidean distance between the location of the reappearing feature point and its last valid occurrence location. Based on the calculated feature recovery index, the occlusion state and reflective instability state of feature points are determined. Specifically, when the feature recovery index is greater than the preset recovery threshold, the corresponding feature point is determined to be a short-term occlusion feature point; when the feature recovery index is lower than the preset recovery threshold and cannot be recovered within the preset time window, the corresponding feature point is determined to be a dynamic reflection instability feature point. The preset recovery threshold is set using an empirical statistical method. By collecting historical image sequence data from typical cultural and tourism scenarios, the feature recovery index distributions corresponding to real short-term occlusion feature points and dynamic reflection instability feature points are calculated respectively. The crossover point of the recovery index distributions of the two types of samples or the critical value that minimizes the classification error is used as the preset recovery threshold.

[0047] Based on the judgment results, the number of effective tracking frames of dynamic reflective instability feature points in continuous image frames is accumulated and statistically analyzed to generate the life cycle parameters of the feature points. The lifetime parameter characterizes the degree of stable persistence of feature points after the removal of dynamic reflection instability interference. The effective tracking frame count refers to the number of image frames in which the feature points can be successfully detected, continuous frame feature matching is completed, and they are not identified as being in a dynamic reflection instability state.

[0048] like Figure 3 As shown, this is a comparison of the decay curves of the lifetime parameter of dynamic reflection instability feature points and short-term occlusion feature points over time: the horizontal axis represents the time series of continuous image frames, and the vertical axis represents the normalized lifetime parameter of the feature points. Figure 3As can be observed, while the lifetime parameter of short-term occluded feature points decreases briefly after temporary occlusion, it quickly recovers and maintains a high level of stability after the occlusion is removed and a stable match is re-established. In contrast, the lifetime parameter of dynamically reflective unstable feature points continuously decays over time due to dynamic reflection interference such as glass reflection and changes in the brightness of mirrors, and is difficult to recover to a stable level. The significant differences in the temporal trends of the two types of feature points indicate that the lifetime parameter construction method based on the feature recovery index proposed in this invention can effectively distinguish between short-term occluded features and dynamic reflective pseudo-features, thereby improving the accuracy of feature point stability assessment in visual SLAM systems.

[0049] It should be noted that the above-mentioned technical solution, which distinguishes between short-term occlusion and dynamic reflective instability based on the feature recovery index and further generates lifecycle parameters, is superior to the traditional lifecycle calculation method that relies solely on the number of consecutive matching frames. This method can perform secondary correction on the true state of feature points after they disappear by using spatial recovery capability and matching confidence. This effectively eliminates the interference of pseudo-instability features caused by glass display case reflections on the statistical results. At the same time, it avoids misjudging temporary loss caused by short-term occlusion as feature failure. This allows the lifecycle parameters to more realistically reflect the ability of feature points to persist under stable and observable conditions, thereby improving the accuracy of feature stability assessment and enhancing the localization robustness and map construction reliability of visual SLAM in complex cultural and tourism reflective environments.

[0050] S4. Based on the position drift rate of the highlighted area and the lifetime parameters of the feature points, identify dynamic reflective pseudo-feature points and reduce the matching weight of the dynamic reflective pseudo-feature points in visual SLAM pose estimation.

[0051] In this embodiment, dynamic reflective pseudo-feature points are identified based on the positional drift rate of the bright area and the lifecycle parameters of the feature points, including: The positional drift rate, feature point lifetime parameters, and feature point matching stability parameters of the highlighted region are obtained to generate an observation feature set. The feature point matching stability parameter is a numerical value obtained by quantifying the reliability of feature point matching between consecutive image frames. Specifically, it is obtained by calculating the ratio of the number of successful matches of the feature point to the total number of matching attempts within a preset time window to obtain the matching success rate. At the same time, the mean and fluctuation of the descriptor distance (Hamming distance when it is a binary descriptor, and Euclidean distance when it is a floating-point descriptor) corresponding to each match are calculated. Based on this, the matching success rate and distance fluctuation are weighted and fused or normalized to obtain the parameter value used to characterize the matching consistency and stability of the feature point in consecutive image frames.

[0052] Based on the set of observed features, a probabilistic graphical model of dynamic reflection state is constructed, in which dynamic reflection state is defined as latent variable, and position drift rate, lifetime parameter and matching stability parameter are defined as observed variable, and a conditional probability relationship between latent variable and observed variable is established. The probabilistic graphical model is either a Bayesian network structure or a factor graph structure.

[0053] Based on the established probabilistic graphical model, probability density mapping is performed on each observation feature, and the conditional probability value under dynamic reflective conditions is calculated to generate the local posterior probability corresponding to each single observation feature. The conditional probability value is calculated based on a preset Gaussian distribution model or kernel density estimation model after normalizing the observed features.

[0054] Based on the obtained local posterior probabilities, the probability results corresponding to different observation features are weighted and fused to generate a comprehensive posterior probability of dynamic reflection state. The weight of each observation feature is predetermined by statistical learning method according to the contribution of each feature to the reflection recognition accuracy in historical labeled samples, and remains fixed or is updated periodically during operation. The weighted fusion process employs a multiplicative Bayesian fusion model, expressed as follows: ; In the formula, Let be the posterior probability of the dynamic reflective state R given the observed feature set X, used to characterize the final confidence level that the current bright region belongs to the dynamic reflective state, where A higher value indicates a higher probability of dynamic reflection. Let i be the i-th observed feature variable. This is the prior probability, used to describe the basic probability distribution of a highlighted area being in a dynamic reflective state before the introduction of current observation data. Let i be the conditional probability of the occurrence of the i-th observed feature under dynamic reflective conditions. Let be the weight coefficient of the i-th observed feature. This is a joint fusion operator for all observed features, used to jointly model multiple conditional probabilities under the Bayesian assumption, reflecting the comprehensive discrimination result under the combined effect of multiple features, where n represents all observed feature variables.

[0055] Based on the generated comprehensive posterior probability, the maximum posterior probability criterion is used to determine the state of the bright area. When the comprehensive posterior probability is greater than the preset determination threshold, the corresponding bright area is determined to be a dynamic reflective area; otherwise, it is determined to be a non-dynamic reflective area. The preset judgment threshold is determined by statistical analysis of the classification results of dynamic reflective areas and non-dynamic reflective areas in historical cultural and tourism scene image data, and the probability value corresponding to the minimum classification error is taken.

[0056] Based on the state determination results, feature points located within the dynamic reflection area are marked, and a set of dynamic reflection pseudo feature points is output.

[0057] Furthermore, reducing the matching weight of the dynamic reflective pseudo-feature points in visual SLAM pose estimation includes: Obtain the set of dynamic reflective pseudo-feature points, and extract the feature descriptors and their initial matching weights of the dynamic reflective pseudo-feature points in the current image frame; The initial matching weights are weight values ​​that the visual SLAM system presets during the feature extraction stage based on feature point response intensity, pyramid level information, and historical matching success rate through a linear weighting method.

[0058] Obtain the matching confidence score for each pseudo-feature point; Based on the obtained matching confidence, the original matching weights of the dynamic reflective pseudo-feature points are adaptively corrected to generate a weight attenuation coefficient. The lower the matching confidence, the smaller the corresponding weight attenuation coefficient. The weight decay coefficient is calculated using the following function: ; In the formula, This is the weight decay coefficient. is the adjustment coefficient, a positive real number used to control the attenuation sensitivity.

[0059] The adjustment coefficient is set using an empirical proportional quantification method, specifically determined based on the ratio between the average brightness value of the dynamic reflective area in the historical cultural and tourism scene image and the average brightness value of the overall image. In one possible implementation, when the ratio of the average brightness value of the dynamic reflective area to the average brightness value of the overall image is between 1.2 and 2.0, the adjustment coefficient can be set to a value between 1 and 5, and mapped according to a linear proportional method. For example, when the brightness ratio is 1.2, the coefficient is set to 1; when the brightness ratio is 2.0, the coefficient is set to 5; and in other cases, a linear interpolation method is used to determine the coefficient.

[0060] Based on the generated weight decay coefficient, the initial matching weight of the dynamic reflective pseudo-feature points is updated and calculated to obtain the corrected matching weight; the corrected matching weight is used to reduce the influence of such feature points in the visual SLAM pose optimization process. The corrected matching weight is calculated by multiplying the initial matching weight by the weight attenuation coefficient, in order to reduce the contribution of dynamic reflective pseudo-feature points to pose estimation.

[0061] The corrected matching weights are input into the pose estimation module of visual SLAM. In the graph optimization process based on minimizing reprojection error, weak constraints are applied to the dynamic reflection pseudo-feature points to obtain a weighted projection error model, so as to reduce its interference on the camera pose estimation results. The weighted projection error model is specifically as follows: ; In the formula, This represents the total weight projection error. The corrected matching weight for the i-th feature point. Let be the observed coordinates of the i-th feature point in the image plane. Let T be the projection function, and T be the camera pose. Let i be the i-th 3D map point.

[0062] Based on the pose optimization results, the global map is updated, and dynamic reflective pseudo-feature points with weights lower than the preset weight threshold are marked as low-confidence feature points.

[0063] The preset weight threshold is obtained by the following quantization method: the initial matching weights of all feature points in 500 consecutive frames of images of the visual SLAM system in a typical scene without significant reflection interference are collected, the 10th percentile is calculated, and this value is used as the preset weight threshold.

[0064] It should be noted that by introducing the high-brightness region drift rate, feature point lifecycle parameters, and matching stability parameters, and using a probabilistic graphical model to perform Bayesian fusion judgment on multi-source observation features, high-confidence recognition of dynamic reflective regions is achieved. Compared with traditional single threshold or rule-based judgment methods, this can effectively reduce misjudgments caused by reflection, occlusion, and low-texture interference. At the same time, during the SLAM pose optimization process, adaptive weight decay and weak constraint processing are applied to the dynamic reflective pseudo-feature points, which significantly reduces their impact on reprojection error optimization. This improves the stability, robustness, continuity, and accuracy of visual SLAM localization in complex cultural and tourism scenarios.

[0065] S5, performs visual SLAM spatial localization based on the remaining feature points after weight adjustment, and performs spatial anchoring and virtual-real fusion display of virtual tour content based on the localization results, including: Obtain the set of remaining feature points after weight adjustment, wherein the set of remaining feature points is the set of effective feature points after removing or reducing the weight of dynamic reflective pseudo feature points; Each feature point includes at least image coordinates, a 3D map point index, and a corrected matching weight.

[0066] Based on the remaining feature point set, establish the correspondence between the two-dimensional image observation points and the three-dimensional spatial map points, and construct the feature constraint set for the current frame; The set of feature constraints is used to describe the observation consistency relationship of three-dimensional spatial points under the current camera imaging conditions.

[0067] Based on the set of feature constraints and combined with the weighted projection error model, the nonlinear least squares optimization method is used to solve the camera pose to obtain the camera pose matrix corresponding to the current image frame. The optimization process is implemented using the Gauss-Newton method or the Levenberg-Marquardt algorithm. Based on the obtained camera pose matrix, the 3D anchor points of the virtual tour content are transformed from the world coordinate system to the camera coordinate system; The virtual tour content includes at least three-dimensional models, text annotations or icon information, and is stored in the world coordinate system in the form of three-dimensional anchor points.

[0068] Based on the coordinate transformation results, perspective projection processing is performed on the virtual guide content to map the three-dimensional anchor points to the two-dimensional image plane of the current image frame, generating virtual overlay display coordinates; Based on the generated virtual overlay display coordinates, combined with image depth information, a virtual-real fusion rendering process is performed to generate a virtual-real fusion display image, so as to achieve spatial alignment display of virtual content and real scene; Based on the fusion of virtual and real images, the spatial anchor points of the virtual tour content are continuously updated frame by frame, and the virtual content is reprojected and adjusted in real time according to changes in camera pose, so as to achieve stable cross-frame following display.

[0069] Furthermore, based on the generated virtual overlay display coordinates and combined with image depth information, a virtual-real fusion rendering process is performed to generate a virtual-real fusion display image, including: Obtain the set of coordinates for the virtual overlay display, and obtain the depth image data corresponding to the current image frame; The virtual overlay display coordinate set is a two-dimensional pixel coordinate set formed after perspective projection transformation, and it corresponds one-to-one with the current image frame in pixel space; Based on the current image frame, a combined luminance statistics and local gradient analysis method is used to detect bright areas and generate a probability map of reflective areas. The calculation method for the reflective region probability map is as follows: the image pixel brightness is normalized and then weighted and fused with the local contrast change rate to obtain the reflective probability value. Regions with a reflective probability value greater than a preset probability threshold are identified as reflective regions. The weighted fusion adopts a linear weighted summation method, where the weight of the brightness normalization value is 0.6 and the weight of the local contrast change rate is 0.4 (the weight allocation is determined by principal component analysis or least squares fitting based on the distinguishing ability of brightness and contrast in historical reflective samples). The preset probability threshold is selected by ROC curve to achieve a balance between the accuracy and recall of reflective detection (e.g., 0.5).

[0070] Based on the probability map of reflective areas, a pixel-by-pixel confidence mapping is performed on the depth image data to generate a depth confidence weight map. The depth reliability weight refers to the normalized coefficient obtained by linearly attenuating the depth reliability based on the reflection probability value corresponding to the pixel position. It is calculated by weighting the reflection probability with a preset attenuation coefficient and then mapping it to the interval (0,1) to characterize the reliability of the pixel depth information.

[0071] Based on the generated depth confidence weight map, the original depth values ​​are corrected pixel by pixel to generate a confidence depth map; The correction method refers to performing pixel-by-pixel multiplication of the original depth value with the depth confidence weight of the corresponding pixel position to attenuate and correct the depth information and generate a reliable depth value.

[0072] Based on the virtual overlay display coordinates, the depth values ​​of the corresponding pixel positions are extracted from the reliable depth map, and a set of spatial differences between the virtual object and the real scene is constructed. Based on the aforementioned set of spatial differences, hierarchical occlusion determination is performed on the virtual objects; Specifically, if the spatial difference does not exceed a preset first spatial tolerance threshold, it is determined to be in a fully visible state; if the spatial difference is greater than the preset first spatial tolerance threshold but does not exceed a preset second spatial tolerance threshold, it is determined to be in a partially visible state; if the spatial difference exceeds the preset second spatial tolerance threshold, it is determined to be in an occlusion state. The preset first spatial tolerance threshold is less than the preset second spatial tolerance threshold.

[0073] The method for obtaining the spatial tolerance threshold is as follows: by collecting historical calibration data of virtual objects and real scenes in typical cultural and tourism scenarios, statistically analyzing the acceptable error range of the virtual rendering results under different depth difference conditions, and taking the mean or quantile of the depth difference corresponding to the boundary point between fully visible and partially visible states as the first spatial tolerance threshold and the second spatial tolerance threshold.

[0074] Based on the occlusion determination result, the transparency weight is adjusted on the virtual object to generate the first virtual rendering layer; The process of adjusting the transparency weight of virtual objects involves using preset transparency baseline coefficients corresponding to three states: fully visible, partially visible, and occluded. These coefficients are combined with the depth difference between the virtual object and its corresponding pixel position, as well as the depth confidence weight. A preset continuous mapping function is then used to calculate a pixel-by-pixel transparency weight value. Finally, the pixel colors of the virtual object are weighted according to the transparency weight value to generate a first virtual rendering layer that is consistent with the real scene space and has a smooth boundary transition effect. This avoids display abrupt changes or occlusion discontinuities caused by relying solely on discrete states.

[0075] The continuous mapping function is the Sigmoid function, which is used to map the depth difference and depth confidence weight to a continuously changing transparency weight parameter.

[0076] The specific calculation formula for the continuous mapping function is as follows: ; In the formula, This is the transparency weight value of the virtual object pixel, used to characterize the visibility of the current virtual pixel during the virtual-real fusion display process. The value ranges from 0 to 1, where a value closer to 1 indicates that the virtual object is more clearly visible, and a value closer to 0 indicates that the virtual object is more transparent. This represents the spatial depth difference between the virtual object and the real scene. The depth difference tolerance threshold is set by collecting historical depth difference data between virtual objects and real scenes in typical cultural and tourism scenarios, and statistically analyzing the average depth difference corresponding to significant occlusion changes of virtual objects in consecutive image frames. This threshold controls the critical range for virtual objects to change from a visible to an occluded state. g is the transparency change adjustment coefficient, set empirically. Specifically, it can be preset to a constant between 2 and 10 based on the display stability requirements of virtual objects in cultural and tourism scenarios. This coefficient controls the steepness of the Sigmoid function curve; a larger g value results in more sensitive transparency changes, while a smaller g value results in smoother transparency changes. This is the depth confidence weight, used to characterize the confidence level of the depth information at the current pixel location.

[0077] Based on the generated first virtual rendering layer, pixel-level fusion processing is performed with the current image frame to generate a virtual-real fused display image; Based on the obtained virtual-real fusion display image, an exponential sliding temporal filtering process is performed on the fusion result between consecutive image frames to suppress flickering caused by dynamic changes in reflective areas.

[0078] The filtering process is as follows: ; In the formula, This is the image output after virtual-real fusion processing at the current time (frame t), used to characterize the stable display image of the current frame. This is the original fused image of the t-th frame at the current time, without temporal filtering. To represent the stable output image result after filtering at the previous time step (frame t-1), and to provide temporal continuity constraints, β is the temporal fusion coefficient, used to control the fusion ratio of current frame information and historical frame information, and its value range is 0≤β≤1.

[0079] It should be noted that, compared to conventional virtual-real fusion methods based on "fixed depth + binary occlusion judgment + simple alpha fusion", the innovation of this technical solution lies in the introduction of a "reflection area-driven depth credibility attenuation mechanism". This means that depth information is no longer used as a uniform and reliable input, but is adaptively corrected pixel by pixel based on the reflection probability in the image. This transforms the originally easily distorted depth data into a reliable depth field. Furthermore, it combines a weighted occlusion model based on a continuous mapping function to achieve a shift from "discrete occlusion judgment" to "continuous visibility modeling". The technical effect of this innovation is that it can significantly reduce depth misjudgment and clipping phenomena in complex scenes such as glass display cases and high-brightness reflections. At the same time, it can effectively improve the smoothness of virtual content boundaries and cross-frame stability, reduce flickering and jitter, thereby improving the spatial alignment accuracy and visual consistency of augmented reality guide systems in real cultural and tourism environments.

[0080] Reference Figure 2 As shown in the diagram, the AR tour guide virtual-real fusion system for cultural and tourism scenarios provided by this invention includes a highlight area and feature point extraction module, a dynamic reflective area judgment module, a pseudo feature point recognition module, and an adaptive SLAM localization and virtual-real fusion module. The modules are interconnected. The highlight area and feature point extraction module is used to collect environmental image frames in cultural and tourism scenes and extract the highlight areas and scene feature points in each image frame. The dynamic reflective region determination module is used to jointly determine the dynamic reflective region based on the positional drift rate and texture stability of the bright region in consecutive image frames. The pseudo-feature point identification module is used to perform time-series tracking of feature points in the dynamic reflective area, calculate their life cycle parameters, and identify dynamic reflective pseudo-feature points based on the position drift rate and life cycle parameters. The adaptive SLAM localization and virtual-real fusion module is used to reduce the matching weight of dynamic reflective pseudo-feature points in visual SLAM pose estimation, use the remaining feature points for spatial localization, and perform spatial anchoring and virtual-real fusion display of virtual tour content.

[0081] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0082] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0084] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An AR-guided virtual-real fusion method for cultural and tourism scenarios, characterized in that, include: Collect environmental image frames in cultural and tourism scenes, and extract the highlighted areas and scene feature points of each image frame; Dynamic reflective areas are jointly determined based on the positional drift rate and texture stability parameters of the bright areas in consecutive image frames. The feature points within the dynamic reflective area are time-series tracked, their lifetime parameters are calculated, and dynamic reflective pseudo-feature points are identified based on the position drift rate and lifetime parameters. Reduce the matching weight of dynamic reflective pseudo-feature points in visual SLAM pose estimation, use the remaining feature points for spatial localization, and perform spatial anchoring and virtual-real fusion display of virtual tour content; The method for obtaining the position drift rate includes: Obtain the detection results of bright regions in consecutive image frames, perform inter-frame region similarity matching based on the parameters of each bright region, and establish the association relationship between bright regions in consecutive frames. The change in center coordinates of the same bright region between adjacent image frames is calculated based on the correlation relationship, and the position drift rate of the bright region is calculated in combination with the inter-frame time interval. The position drift rate is used to characterize the degree of position change of the bright region per unit time. The jointly determined dynamic reflective area includes: Extract local texture features from the highlighted areas in consecutive image frames to generate texture feature vectors; The similarity is calculated based on the texture feature vectors of the corresponding highlighted areas in the current frame and the previous frame to obtain the texture stability parameter, which is used to characterize the stability of the texture structure inside the highlighted area. The determination is made jointly based on the position drift rate and the texture stability parameter: if the position drift rate is higher than a preset drift rate threshold and the texture stability parameter is lower than a preset texture stability threshold, then it is determined to be a dynamic reflective area. Output the position parameters and texture stability parameters of the dynamic reflective area; The step of performing time-series tracking of feature points within the dynamic reflective area and calculating their lifecycle parameters includes: Feature points located within and around the dynamic reflective area are selected, and feature descriptor matching is performed between consecutive frames to establish the temporal correspondence of the same feature point. The feature points are tracked continuously frame by frame, and their appearance, disappearance and reappearance states are recorded; The feature recovery index is calculated based on whether the feature points are recovered within a preset time window after they disappear, as well as the spatial offset and matching confidence during recovery. Based on the feature recovery index, the feature point is determined to be a short-term occlusion feature point or a dynamic reflection instability feature point; The number of effective tracking frames in consecutive frames is counted for dynamic reflective instability feature points, and feature point lifetime parameters are generated.

2. The method according to claim 1, characterized in that, The process of collecting environmental image frames from cultural and tourism scenes and extracting highlight areas and scene feature points from each image frame includes: Continuously acquire environmental images of cultural and tourism scenes to generate environmental image frame sequences; Each image frame is preprocessed, and a brightness threshold is dynamically determined based on the brightness distribution of the image frame. Highlighted areas with brightness exceeding the threshold are then extracted. Perform connected component analysis on the highlighted region to obtain the parameters of the highlighted region; Extract scene feature points from non-highlighted areas in the image frame, and label feature points located within or near the edges of highlighted areas as highlighted associated feature points; Output the results of highlight region detection and scene feature point extraction in the current image frame.

3. The method according to claim 1, characterized in that, The identification of dynamic reflective pseudo-feature points based on the position drift rate and lifecycle parameters includes: Obtain the position drift rate of the highlighted area, the lifetime parameters of the feature points, and the matching stability parameters, and construct an observation feature set; Based on the set of observed features, a probabilistic graphical model of dynamic reflection state is established, each observed feature is mapped to a local posterior probability, and weighted fusion is performed to obtain the comprehensive posterior probability of dynamic reflection state. Based on the comprehensive posterior probability, it is determined whether the bright area is a dynamic reflective area, and the feature points located in the area are marked as dynamic reflective pseudo-feature points.

4. The method according to claim 1, characterized in that, The reduction of the matching weight of dynamic reflective pseudo-feature points in visual SLAM pose estimation includes: Obtain the initial matching weights and matching confidence scores of dynamic reflective pseudo-feature points; Based on the matching confidence, a weight decay coefficient is generated, and the initial matching weight is adaptively corrected to obtain the corrected matching weight. The corrected matching weights are input into the visual SLAM pose estimation module, and weak constraints are applied to the dynamic reflective pseudo-feature points during the reprojection error optimization process to reduce their interference with the pose estimation results.

5. The method according to claim 1, characterized in that, The method of using remaining feature points for spatial positioning and spatially anchoring and fusion display of virtual tour content includes: Obtain the set of remaining feature points after weight adjustment, and establish the correspondence between two-dimensional image observation points and three-dimensional spatial map points; Based on the aforementioned correspondence and the weighted projection error model, nonlinear least squares optimization is used to solve the camera pose of the current frame. Based on the obtained camera pose, the 3D anchor points of the virtual tour content are transformed from the world coordinate system to the camera coordinate system, and then mapped onto the 2D image plane through perspective projection to generate a virtual-real fusion display image. Based on the continuous changes in camera pose, the spatial anchor points of the virtual tour content are reprojected and adjusted in real time to achieve stable cross-frame following display.

6. The method according to claim 5, characterized in that, The generation of the virtual-real fusion display image includes: Obtain the coordinates of the virtual overlay display and the depth image data of the current frame; Detect reflective areas in the current image frame, generate a reflective area probability map, and perform pixel-by-pixel confidence mapping and correction on the depth image based on this map to obtain a reliable depth map. Depth values ​​are extracted from the reliable depth map based on the coordinates of the virtual overlay display, the spatial difference between the virtual object and the real scene is constructed, and hierarchical occlusion judgment is performed based on the preset spatial tolerance threshold. Based on the occlusion determination results, combined with the depth difference and depth confidence weight, the transparency of the virtual object is continuously adjusted to generate a rendering layer and blend it with the current image frame to obtain a virtual-real fusion display image; Temporal filtering is applied to the fusion results of consecutive frames to suppress display flicker caused by dynamic changes in reflective areas.

7. A system using the AR-guided virtual-real fusion method for cultural and tourism scenarios as described in any one of claims 1-6, characterized in that, include: The highlight area and feature point extraction module is used to collect environmental image frames in cultural and tourism scenes and extract the highlight areas and scene feature points in each image frame. The dynamic reflective area determination module is used to jointly determine the dynamic reflective area based on the position drift rate and texture stability parameters of the bright area in consecutive image frames. The pseudo-feature point identification module is used to perform time-series tracking of feature points in the dynamic reflective area, calculate their life cycle parameters, and identify dynamic reflective pseudo-feature points based on the position drift rate and life cycle parameters. The adaptive SLAM localization and virtual-real fusion module is used to reduce the matching weight of dynamic reflective pseudo-feature points in visual SLAM pose estimation, use the remaining feature points for spatial localization, and perform spatial anchoring and virtual-real fusion display of virtual tour content.

Images