A Multi-Scenic Area AR Interaction System and Method with Cross-Device Pose Collaboration and Occlusion Perception

Abstract

This invention relates to the field of multi-device collaborative positioning and rendering technology, and discloses a multi-scene AR interaction system and method with cross-device pose collaboration and occlusion perception. The system includes: a heterogeneous data acquisition module that collects sensor data from mobile and wearable devices; a statistical fusion module that uses a covariance intersection algorithm to fuse heterogeneous poses and error covariance to minimize uncertainty; an environment perception module that extracts viewport texture entropy and environmental semantic edges; a potential energy field correction module that constructs an anisotropic potential energy field based on Mahalanobis distance and dynamically adjusts the stiffness coefficient in combination with texture entropy to correct pose drift; and a rendering interaction module that projects the 3D positioning uncertainty onto a 2D screen, performs probabilistic soft occlusion rendering, and constructs anisotropic interaction hotspots. This invention solves the registration problem caused by the cross-correlation of heterogeneous positioning errors and anisotropic distribution, significantly improving the accuracy and robustness of virtual-real fusion in high-dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of multi-device collaborative positioning and rendering technology, specifically to a multi-scenic area AR interactive system and method with cross-device pose collaboration and occlusion perception. Background Technology

[0002] With the increasing application of augmented reality (AR) technology in outdoor cultural tourism and large-scale guided tours, tourists often use handheld mobile terminals and head-mounted AR display devices simultaneously during their visits. These devices typically each carry a global navigation satellite system, inertial measurement unit, or visual sensor, possessing independent self-positioning capabilities. However, limited by the hardware cost and power consumption control of consumer-grade devices, a single device struggles to maintain high-precision positioning over long distances or in areas with uniform environmental characteristics, easily leading to cumulative drift. While existing multi-source information fusion technologies attempt to improve overall positioning accuracy by combining data from different devices, traditional methods often rely on the assumption that the observation noise of each sensor is independent when processing collaborative calculations between mobile and wearable data. In actual physical space, multiple devices are in the same environmental field, often observing the same visual feature points or experiencing similar environmental interference, resulting in difficult-to-quantify cross-correlation between the observation errors of each device. In this situation, directly using conventional Kalman filtering or weighted averaging methods for fusion can lead to an overly optimistic estimated state covariance matrix, failing to accurately reflect the confidence level of the current positioning. In severe cases, it can even cause filter divergence, resulting in drastic changes in virtual content or positioning loss.

[0003] To correct sensor drift, existing augmented reality systems typically use pre-built digital maps for geometric registration with landmarks in the real environment. Most current registration correction algorithms rely on Euclidean distance to search for the nearest matching point between the virtual location and the real landmark. This geometric distance-based correction logic ignores the anisotropic nature of sensor positioning errors in spatial distribution. For example, satellite positioning systems typically have much larger errors in the vertical direction than in the horizontal direction, and visual odometry often has higher uncertainty in the depth direction than in the imaging plane direction. Forcing alignment solely based on Euclidean distance without considering these statistical differences often introduces new biases while correcting drift, compromising the previously high accuracy in certain dimensions. Furthermore, existing pose smoothing algorithms often use fixed filtering parameters or stiffness coefficients, failing to fully utilize the psychophysical characteristics of the human visual system. The human eye is less sensitive to image jitter against textured backgrounds but extremely sensitive to jitter against flat backgrounds. Existing fixed-parameter methods cannot dynamically adjust the correction strategy based on background texture features, making it difficult to achieve a good balance between suppressing jitter and ensuring motion tracking.

[0004] Positioning uncertainty not only affects the accuracy of virtual-real overlay but also directly interferes with graphics rendering quality and human-computer interaction experience. In the presence of high-frequency positioning jitter, traditional graphics rendering pipelines employ deterministic depth buffer testing, which can lead to depth conflicts between virtual objects and real occlusions at the edges, resulting in edge flickering or incorrect occlusion relationships, thus disrupting the immersive experience of augmented reality. Simultaneously, at the human-computer interaction level, positioning jitter causes virtual points of interest to sway irregularly in screen space. Existing interaction decision logic is typically based on fixed static collision bounding boxes, failing to consider the projection distribution of positioning errors on the screen. This makes it difficult for users to accurately click or select targets in dynamic scenes, significantly reducing the success rate of interactive operations and system usability. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multi-scenic area AR interactive system and method with cross-device pose collaboration and occlusion perception. It solves the problems of low virtual-real registration accuracy caused by ignoring error cross-correlation and anisotropic distribution in heterogeneous multi-source positioning fusion of existing augmented reality systems, as well as the lack of visual rendering and interactive adaptive compensation mechanisms for positioning jitter.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-scenic area AR interactive system and method with cross-device pose collaboration and occlusion perception, comprising:

[0007] The heterogeneous data acquisition module is configured to acquire sensor data from mobile computing devices and augmented reality display devices, and output the state estimates of each device in the local coordinate system and the corresponding state error covariance matrix.

[0008] The statistical fusion module is connected to the heterogeneous data acquisition module and is configured to receive the pose state and state error covariance matrix of multi-source heterogeneous data, perform data fusion calculation using the covariance intersection algorithm, and output the fused pose estimate and the fusion covariance matrix representing the confidence interval of the position estimate.

[0009] The environment perception module is configured to extract the real background image within the viewport and calculate the viewport texture entropy, while simultaneously acquiring geometric edge features in the real environment to generate a semantic edge set.

[0010] The potential energy field correction module is connected to the statistical fusion module and the environment perception module. It is configured to construct a spatial Mahalanobis distance field using the fusion covariance matrix, search for the real edge point that minimizes the Mahalanobis distance in the semantic edge set as the anchor point, and calculate the nonlinear offset vector of the virtual object relative to the real edge to output the corrected pose.

[0011] The rendering interaction module is connected to the potential field correction module and is configured to generate a screen error ellipse and depth variance using the fusion covariance matrix, calculate the visibility probability based on the depth variance to perform probabilistic soft occlusion rendering, and generate an anisotropically expanded interactive hotspot using the screen error ellipse.

[0012] Preferably, the statistical fusion module is specifically configured as follows:

[0013] The system receives the mobile terminal state vector and its covariance matrix, as well as the wearable terminal state vector and its covariance matrix, output by the heterogeneous data acquisition module, and performs interpolation and alignment on the heterogeneous data based on the timestamp.

[0014] The hand-eye extrinsic parameter matrix is ​​used to transform data from local coordinate systems of different devices to a unified fusion coordinate system, and a similarity transformation based on the rotation matrix is ​​performed on the state error covariance matrix.

[0015] Define an optimization objective function to find the optimal weighting coefficients that minimize the trace of the fused state error covariance matrix;

[0016] Based on the optimal weighting coefficients, the information matrices of the aligned mobile and wearable terminals are weighted, summed, and inverted to obtain the fused state error covariance matrix.

[0017] Preferably, the environmental perception module is specifically configured as follows:

[0018] Gradient statistical analysis is performed on the image data within the viewport at the current moment to construct a normalized gradient magnitude histogram, and the viewport texture entropy is calculated using the Shannon entropy formula to characterize the texture complexity.

[0019] Communicate with edge computing nodes to upload the current fusion location and pull surrounding lightweight vector map data;

[0020] The vector map data is subjected to view frustum culling, and the selected three-dimensional vector features are transformed from the geodetic coordinate system to the local fusion coordinate system to generate the semantic edge set.

[0021] Preferably, the potential energy field correction module includes a potential energy field construction unit, which is configured as follows:

[0022] Extract the position error covariance matrix from the fused covariance matrix;

[0023] A scalar potential field based on Mahalanobis distance is constructed using the position error covariance matrix. The scalar potential field defines the weighted statistical distance of any point in space relative to the current fused position estimate.

[0024] Based on the eigenvalue decomposition characteristics of the position error covariance matrix, the scalar potential field adaptively weights the spatial error, assigning a lower distance gradient in directions with higher sensor uncertainty to allow for a larger range of corrections, while strictly constraining the position correction amount in directions with lower uncertainty.

[0025] Preferably, the potential energy field correction module further includes an anchor point matching unit, which is configured as follows:

[0026] The geometric elements in the semantic edge set are traversed to solve an optimization problem to determine the optimal anchor point; the optimization problem is to find the geometric feature point that minimizes the Mahalanobis distance.

[0027] When the elements in the semantic edge set are line segments, the optimal anchor point is the projection point on the line segment that minimizes the quadratic function value corresponding to the Mahalanobis distance. This optimal anchor point is located in the direction with the greatest system positioning uncertainty.

[0028] Preferably, the potential field correction module further includes a parameter coupling unit, which is configured as follows:

[0029] The trace of the fused covariance matrix is ​​calculated to characterize the overall positioning error energy; the dynamic stiffness coefficient is calculated according to the exponential coupling model, which determines the weight strength of the system in correcting the position of virtual objects based on semantic anchors;

[0030] The dynamic stiffness coefficient is calculated as follows: based on the baseline stiffness coefficient, multiplied by an enhancement term constructed from the uncertainty gain factor and the trace of the fusion covariance matrix, and then multiplied by an exponential decay term constructed from the texture suppression attenuation factor and the viewport texture entropy.

[0031] Preferably, the potential energy field correction module further includes a nonlinear offset correction unit, which is configured as follows:

[0032] Calculate the original deviation vector from the current fusion position to the optimal anchor point; combine the dynamic stiffness coefficient and the time step to calculate the instantaneous correction increment within a single frame;

[0033] A nonlinear saturation function is introduced to limit the instantaneous correction increment, and truncation is performed when the input vector magnitude exceeds a preset threshold.

[0034] The cumulative position offset is updated using a drift accumulation model, and this offset is superimposed on the fused position to generate the virtual camera position;

[0035] The rendering pose is calculated using spherical linear interpolation, and a view matrix is ​​constructed based on the transpose of the rotation matrix of the rendering pose and the position of the virtual camera.

[0036] Preferably, the rendering interaction module includes a projection transformation unit and a probabilistic rendering unit:

[0037] The projection transformation unit is configured to use the Jacobian matrix of the perspective projection function to propagate the position covariance matrix in the camera coordinate system to the two-dimensional screen space to generate the screen space covariance matrix, and extract the variance components in the corresponding depth direction as depth uncertainty parameters.

[0038] The probabilistic rendering unit is configured to sample the environmental depth texture to obtain the real scene depth value, and use the standard form of the Gaussian cumulative distribution function to calculate the visibility probability of the virtual pixel relative to the real scene.

[0039] The input for calculating the visibility probability includes the difference between the real scene depth value and the expected depth value of the virtual fragment, as well as the depth uncertainty parameter.

[0040] Preferably, the rendering interaction module further includes an interaction determination unit, which is configured as follows:

[0041] The collision bounding box of the virtual object is statistically expanded using the screen space covariance matrix; the Mahalanobis distance between the user touch point on the screen and the center projection point of the virtual object is calculated.

[0042] Using the Mahalanobis distance being less than a preset threshold as a condition for hit determination, an interactive hot zone based on anisotropic error is constructed.

[0043] A multi-scenic area AR interaction method with cross-device pose collaboration and occlusion perception includes the following steps;

[0044] Configure a heterogeneous data acquisition module to collect sensor data from mobile computing devices and augmented reality display devices, and output the state estimates of each device in the local coordinate system and the corresponding state error covariance matrix;

[0045] The configuration statistical fusion module receives the pose state and state error covariance matrix from multiple heterogeneous sources, performs data fusion calculation using the covariance intersection algorithm, and outputs the fused pose estimate and the fusion covariance matrix representing the confidence interval of the position estimate.

[0046] The environment awareness module is configured to extract the real background image within the viewport and calculate the viewport texture entropy, while simultaneously acquiring geometric edge features in the real environment to generate a semantic edge set.

[0047] The configuration potential field correction module uses the fusion covariance matrix to construct a spatial Mahalanobis distance field, searches for the real edge point that minimizes the Mahalanobis distance in the semantic edge set as the anchor point, and calculates the nonlinear offset vector of the virtual object relative to the real edge to output the corrected pose.

[0048] The configuration rendering interaction module uses the fusion covariance matrix to generate a screen error ellipse and depth variance, calculates the visibility probability based on the depth variance to perform probabilistic soft occlusion rendering, and uses the screen error ellipse to generate an anisotropically expanded interactive hotspot.

[0049] This invention provides a multi-scene AR interactive system and method with cross-device pose collaboration and occlusion perception. It has the following beneficial effects:

[0050] 1. This invention employs a covariance intersection algorithm in a statistical fusion module to fuse heterogeneous pose data from mobile and wearable devices, effectively solving the problem of unknown cross-correlation caused by overlapping environmental features in collaborative localization between the two devices. Compared to traditional fusion methods that assume independent errors, this invention optimizes the weighting coefficients to minimize the trace of the fused covariance matrix, significantly reducing the overall localization uncertainty of the system while ensuring statistical consistency of the estimation results. This provides a more accurate and robust pose reference for augmented reality interaction.

[0051] 2. This invention utilizes a potential energy field correction module to construct an anisotropic potential energy field based on Mahalanobis distance, and dynamically adjusts the correction intensity by combining viewport texture entropy. This mechanism leverages the eigenvalue distribution of the position error covariance matrix to perform significant corrections only in directions with high sensor uncertainty using environmental semantic edges, while maintaining observation confidence in directions with low uncertainty. Simultaneously, it automatically enhances the stiffness coefficient in low texture entropy environments, effectively suppressing the visual slippage of virtual objects caused by accumulated sensor errors and improving the stability of virtual-real registration.

[0052] 3. This invention projects the positioning uncertainty in three-dimensional space to two-dimensional screen space in the rendering interaction module, generating a screen error ellipse and depth variance. By calculating probabilistic soft occlusion based on depth variance, the depth conflict jumps caused by positioning jitter at the real occlusion edges of virtual objects are eliminated, improving the visual fusion effect. At the same time, the screen error ellipse is used to anisotropically expand the interaction area, enabling users to accurately hit virtual objects even in high-dynamic scenes where positioning jitter exists, significantly improving the success rate of human-computer interaction. Attached Figure Description

[0053] Figure 1 This is a flowchart of the method of the present invention;

[0054] Figure 2 This is a schematic diagram of the logical structure of the heterogeneous data acquisition module of the present invention;

[0055] Figure 3 This is a schematic diagram of the logical processing flow of the statistical fusion module of the present invention;

[0056] Figure 4This is a schematic diagram of the logic processing flow of the environmental perception module of the present invention;

[0057] Figure 5 This is a schematic diagram of the potential energy field correction module of the present invention;

[0058] Figure 6 This is a schematic diagram illustrating the logical principle of the rendering interaction module of the present invention;

[0059] Figure 7 This is a flowchart of the method of the present invention. Detailed Implementation

[0060] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Please see the appendix Figure 1 This invention provides a multi-scenic area AR interactive system with cross-device pose collaboration and occlusion perception, including:

[0062] The heterogeneous data acquisition module 100 is configured to acquire GNSS data, IMU data and SLAM data from mobile computing devices in real time, and simultaneously acquire VIO data from augmented reality display devices, and output the state estimates of each device in the local coordinate system and the corresponding state error covariance matrix.

[0063] The statistical fusion module 200 is connected to the heterogeneous data acquisition module 100 and is configured to receive the pose state and state error covariance matrix of multi-source heterogeneous data. It performs data fusion calculation using the covariance intersection algorithm, minimizes the trace or determinant of the fused covariance matrix by optimizing the weighting coefficients, and outputs the fused pose estimate and the fused covariance matrix representing the confidence interval of the position estimate.

[0064] The environment perception module 300 is configured to extract the real background image within the viewport and calculate the viewport texture entropy to quantify the background's ability to mask visual jitter. At the same time, it pulls the surrounding lightweight vector map from the server and extracts the significant geometric edge features in the real environment to generate a semantic edge set.

[0065] The potential energy field correction module 400 is connected to the statistical fusion module 200 and the environmental perception module 300. It is configured to construct a spatial Mahalanobis distance field using the fusion covariance matrix with the current fusion position of the virtual object as the center, search for the real edge point that minimizes the Mahalanobis distance in the semantic edge set as the anchor point, and calculate the nonlinear offset vector of the virtual object relative to the real edge to output the corrected pose.

[0066] The rendering interaction module 500 receives the corrected pose, fusion covariance matrix, and viewport texture entropy. It is configured to project the fusion covariance matrix onto the two-dimensional screen space and depth direction to generate screen error ellipses and depth variance. Based on the depth variance, it calculates the visibility probability of virtual pixels relative to the real occlusion depth to achieve probabilistic soft occlusion rendering. It also uses the screen error ellipses to perform Minkowski sum operations on the original colliders of virtual objects to generate anisotropically expanded interactive hot zones.

[0067] The technical principles and execution process of the above modules will be explained in detail below with reference to specific embodiments and accompanying drawings.

[0068] See attached document Figure 2 , Figure 2 This is a schematic diagram of the logical structure of the heterogeneous data acquisition module in this embodiment. The heterogeneous data acquisition module 100 specifically includes a mobile acquisition unit and a wearable acquisition unit. The mobile acquisition unit is deployed on a handheld mobile terminal and is configured to read data from the microelectromechanical system (MEMS) sensor and camera module through a hardware abstraction layer; the wearable acquisition unit is deployed on augmented reality glasses and is configured to read data from its built-in visual inertial module.

[0069] The mobile acquisition unit acquires in real time the geodetic coordinate data output from the GNSS receiver of the mobile terminal, the three-axis acceleration and three-axis angular velocity data output from the IMU, and the image stream data from a monocular or multi-view camera. For the above raw sensor data, the mobile acquisition unit calculates the motion state of the mobile terminal in its independently established local navigation coordinate system using a built-in visual SLAM backend or VIO algorithm thread. The wearable acquisition unit primarily relies on the high-frequency IMU data and grayscale or depth camera data from the augmented reality glasses, utilizing a sliding window optimized or extended Kalman filter (EKF) algorithm to independently calculate the six-degree-of-freedom pose of the wearable device.

[0070] To achieve subsequent statistical fusion, a unified state-space representation of each device needs to be constructed. For any data acquisition terminal in the system... (in The value can be an identifier representing the mobile terminal. Or an identifier representing wearable devices ), which is in time step state vector It is defined as a high-dimensional vector containing three-dimensional position, three-dimensional velocity, attitude quaternions, and sensor zero bias.

[0071] While outputting the state estimate, the heterogeneous data acquisition module 100 simultaneously calculates and outputs the state error covariance matrix, which represents the confidence level of the current state estimate. State error covariance matrix The quantification of the deviation distribution characteristics between the state estimate and the actual state value provides the foundational data for subsequent covariance intersection fusion and potential field construction. State error covariance matrix. This is a symmetric positive definite matrix, where the diagonal elements correspond to the variances of the components of the state vector, and the off-diagonal elements correspond to the covariances between the components. In this embodiment, the variances are specifically extracted... The three-dimensional submatrix corresponding to the position vector is denoted as the position error covariance matrix. The three-dimensional submatrix corresponds to a three-dimensional confidence ellipsoid in geometric space, and the major axis of the ellipsoid indicates the direction of greatest uncertainty in the position estimation.

[0072] See attached document Figure 3 , Figure 3 This is a schematic diagram of the logical processing flow of the statistical fusion module in this embodiment. The statistical fusion module 200 specifically includes a spatiotemporal alignment unit and a covariance intersection calculation unit.

[0073] The spatiotemporal alignment unit is configured to receive the mobile terminal state vector and its covariance matrix output by the heterogeneous data acquisition module 100, as well as the wearable terminal state vector and its covariance matrix. Due to differences in sampling frequencies and data transmission links between different devices, the spatiotemporal alignment unit performs interpolation alignment on the heterogeneous data based on timestamps. Linear interpolation is used for position states, and spherical linear interpolation (Slerp) is used for attitude quaternion states to ensure the normalization characteristics of the attitude interpolation.

[0074] The spatiotemporal alignment unit utilizes a pre-calibrated hand-eye extrinsic parameter matrix to transform data from different device local coordinate systems to a unified fused coordinate system. For the position component in the state vector, a rotation-translation transformation is performed; for the state error covariance matrix, a similarity transformation based on the rotation matrix is ​​performed. Specifically, let the rotation matrix from the local coordinate system to the fused coordinate system be... The local covariance matrix is The transformed covariance matrix Calculated as This step ensures the consistency of the covariance matrices involved in the fusion across the spatial reference.

[0075] The covariance intersection (CI) calculation unit addresses the issue of unknown cross-correlation among multi-source heterogeneous sensors by performing conservative consistency fusion calculations. In scenarios where mobile terminals and augmented reality glasses coordinate localization, although their localization algorithms run independently, they may observe the same environmental feature points or be subject to similar environmental interference, leading to statistically complex correlations between estimation errors. Directly using Kalman filters that assume independent errors for fusion results in estimated variances smaller than actual errors, causing filter divergence. This embodiment employs the covariance intersection (CI) algorithm, which introduces weighted coefficients to construct a convex combination, ensuring statistical consistency in the fused estimation results.

[0076] The covariance intersection calculation unit defines the optimization objective function, aiming to find the optimal weighting coefficients. This makes the fused state error covariance matrix The uncertainty metric is minimized. This uncertainty metric is represented by the trace or determinant of a matrix. In this embodiment, minimizing the trace of the matrix is ​​preferably used as the objective function, and its mathematical expression is:

[0077] ;

[0078] In the formula, This represents the minimize operator. The fusion weights determine the ratio between the two information sources. Indicates the range of values, limiting It must be between 0 and 1. Represents the trace of a matrix. This represents the state error covariance matrix after fusion.

[0079] The covariance intersection calculation unit calculates the moving end covariance matrix based on the coordinates of the fused coordinate system. Covariance matrix of wearable device The optimal weighting coefficients are obtained by solving the above optimization problem using a one-dimensional search algorithm. Based on this optimal weighting coefficient, the fused state error covariance matrix... With fusion state estimate Calculate using the following formula:

[0080] ;

[0081] ;

[0082] In the formula, This represents the fused state error covariance matrix. , This represents the covariance matrix of the state error between the mobile device and the wearable device after coordinate system alignment. , This represents the corresponding information matrix. This represents the fused state estimation vector. , This represents the state estimation vectors of the mobile and wearable devices after coordinate system alignment. This represents the CI fusion weighting coefficient. It was obtained through one-dimensional search optimization. This represents the weighting coefficient corresponding to the wearable device. This represents the matrix inversion operation.

[0083] See attached document Figure 4 , Figure 4 This is a schematic diagram of the logic processing flow of the environment perception module in this embodiment. The environment perception module 300 specifically includes a texture analysis unit and a semantic retrieval unit. These two units work in parallel, providing environmental adaptive parameters for subsequent pose correction and physical reference benchmarks for geometric constraints, respectively. The texture analysis unit is configured to perform gradient statistical analysis on the image data within the AR viewport at the current moment to quantify the masking effect of the environmental background on visual errors. The texture analysis unit receives the current frame image transmitted by the heterogeneous data acquisition module 100, converts it into a grayscale image, and uses the Sobel operator or Prewitt operator to calculate the gradient components of each pixel in the horizontal and vertical directions, thereby synthesizing the gradient magnitude.

[0084] The texture analysis unit statistically analyzes the distribution of gradient magnitudes of all pixels within the current viewport or region of interest (ROI) and constructs a normalized gradient magnitude histogram. Let the discrete order of the gradient magnitudes be... , No. The probability of a gradient magnitude occurring is (If the sum of probabilities is 1), then the viewport texture entropy Calculated using the Shannon entropy formula:

[0085] ;

[0086] In the formula, The texture entropy of the viewport or ROI region is used to characterize texture complexity. This represents the total number of discrete series representing the gradient magnitude. Indicates the gradient magnitude level index. , Indicates the first Gradient magnitude, The gradient magnitude is represented as The probability of a pixel appearing in the ROI satisfies , It represents a logarithmic operation with base 2.

[0087] The output of the texture analysis unit The value is used as a gain factor to adjust the anti-jitter strength of the system and is input to the subsequent potential energy field correction module.

[0088] Higher values This indicates that the current viewport is in a high-masking environment, and is configured to reduce the magnetic stiffness coefficient to prioritize rendering smoothness;

[0089] lower values This indicates that the current viewport is in a low-mask environment and is configured to enhance the magnetic stiffness coefficient to preferentially suppress visual jitter.

[0090] The semantic retrieval unit is configured to establish a geometric relationship between a virtual coordinate system and the real geographic environment. The semantic retrieval unit communicates with edge computing nodes via a wireless network and uploads the fused location output by the current statistical fusion module 200. Edge nodes index a lightweight vector map database stored in the cloud based on their location and then distribute environmental semantic data around the current location.

[0091] The environmental semantic data comprises a 3D vector dataset of salient landmarks, building outlines, and road boundaries in the real physical world. Upon receiving this vector data, the semantic retrieval unit first performs frustum culling based on the current fused pose and camera frustum parameters, filtering out geometric features visible in the current viewport. To address the alignment issue between different coordinate systems, the semantic retrieval unit performs coordinate projection transformation. Specifically, using the transformation matrix between the global geodetic coordinate system and the local inertial coordinate system established during system initialization, or based on the projection rules of the tangent plane (ENU) determined at the current location, the filtered 3D vector features are transformed from the geodetic coordinate system to the system's local fused coordinate system, generating a semantic edge set. .

[0092] Semantic edge set Each element in the image is composed of three-dimensional line segments or discrete point clouds, representing physical edges with strong geometric constraints in the real environment. These real physical edges will serve as magnetic anchor points in subsequent modules to correct registration errors between virtual objects and the real environment caused by sensor drift. Through the above processing, the environment perception module 300 outputs entropy parameters characterizing the visual characteristics of the environment and vector data characterizing the geometric structure of the environment in real time.

[0093] See attached document Figure 5 , Figure 5 This is a schematic diagram of the potential energy field correction module according to an embodiment of the present invention. The potential energy field correction module 400 specifically includes a potential energy field construction unit, an anchor point matching unit, a parameter coupling unit, and a nonlinear offset correction unit. The potential energy field construction unit is configured to map the statistical feature parameters output by the statistical fusion module 200 into a geometric constraint field in three-dimensional space.

[0094] The potential field construction unit receives the fused position estimate output by the statistical fusion module 200. and the position error covariance matrix In heterogeneous sensor systems, the positioning error distribution exhibits significant anisotropy. For example, the variance of GNSS signals in the vertical direction is typically greater than that in the horizontal direction, while the variance of visual SLAM in the depth direction is typically greater than that in the direction parallel to the imaging plane. To accurately reflect this directional difference in pose correction, the potential field construction unit utilizes the position error covariance matrix... Construct a scalar potential energy field based on Mahalanobis distance.

[0095] The scalar potential field defines any point in space. Relative to the current estimated position The weighted statistical distance. Specifically, for any candidate point in three-dimensional space... Its Mahalanobis distance Calculate according to the following quadratic formula;

[0096] ;

[0097] In the formula, Indicate candidate points Compared to the Mahalanobis distance estimated by the current fused location, This represents the coordinate vector of a candidate target point in three-dimensional space. This represents the fused location estimation center of the system at the current moment. This represents the offset vector of the candidate point relative to the fusion position. This represents the position error covariance matrix of the fused position estimate. This represents the inverse matrix of the position error covariance matrix.

[0098] To prevent abnormal sensor data from causing If the potential energy field building unit is singular or near-illness-prone and therefore cannot be inverted, it is configured to perform regularization (e.g., adding a small positive definite diagonal matrix) when calculating the inverse matrix. Alternatively, singular value decomposition (SVD) can be used to calculate the Moore-Penrose pseudoinverse.

[0099] The potential energy field constructed by the above formula has anisotropic geometric characteristics. The equipotential surfaces of the potential energy field are defined as satisfying... All points The set of equipotential surfaces. Geometrically, these equipotential surfaces constitute a set of . A three-dimensional ellipsoid centered on.

[0100] Based on covariance matrix The eigenvalue decomposition characteristics enable the potential field to achieve adaptive weighting for spatial errors.

[0101] In directions corresponding to larger eigenvalues ​​(i.e., directions with higher sensor uncertainty), the distance gradient of the potential energy field is smaller. This means that for the same Euclidean distance offset, the calculated Mahalanobis distance increment is smaller, assigning a lower penalty weight to the geometric deviation in this direction and allowing for a larger range of corrections in this direction.

[0102] In directions corresponding to smaller eigenvalues ​​(i.e., directions with lower sensor uncertainty), the distance gradient of the potential energy field is larger. This means that a small change in Euclidean distance will lead to a sharp increase in Mahalanobis distance, assigning a higher penalty weight to geometric deviations in that direction, thereby strictly constraining the position correction in that direction.

[0103] Through the above processing, the potential field construction unit reconstructs the physical space into a metric space based on statistical standard deviation, ensuring that the subsequent pose correction process strictly follows the Bayesian inference logic, that is, only on dimensions with high prior estimation uncertainty, environmental semantics is used to make significant corrections, while on dimensions with high prior estimation certainty, the weights of sensor observations are maintained.

[0104] Anchor point matching unit is configured as a semantic edge set output by environment perception module 300. In the middle, the search and current fusion location The closest geometric feature point is used as a semantic anchor. Unlike traditional nearest-point iterative algorithms based on Euclidean distance, the anchor point matching unit uses the Mahalanobis distance constructed above during the search process. As a metric.

[0105] Due to semantic edge set Composed of discretized point clouds or continuous line segments, the anchor point matching unit is configured to traverse the geometric elements in the set, and solve the following optimization problem to determine the optimal anchor point:

[0106] ;

[0107] In the formula, Indicates the optimal anchor point coordinates. This represents candidate points or points on line segments within a semantic edge set. This represents a set of semantic edges, consisting of discrete edge points or continuous edge segments. Denotes the independent variable that minimizes the objective function. This represents the fused location estimation center of the system at the current moment. This represents the offset vector of the candidate point relative to the fusion position. This represents the position error covariance matrix of the fused position estimate. This represents the inverse matrix of the position error covariance matrix.

[0108] when When the element in the formula is a line segment, the above formula... This represents the projection point on the line segment that minimizes the value of the quadratic function. The optimization process is geometrically equivalent to finding a point on the line segment that minimizes the value of the quadratic function. The edge points tangent to the minimum envelope confidence ellipsoid centered at the center. Anchor points selected by the anchor point matching unit. Located in the direction of greatest system positioning uncertainty, this ensures that subsequent pose correction operations can specifically suppress spindle drift caused by sensor noise. To improve search efficiency, the anchor point matching unit can pre-calculate based on the fused position. Construct a local bounding box and calculate Mahalanobis distance only for the semantic edges within the bounding box.

[0109] The parameter coupling unit is configured to compute the connection between the virtual object and the semantic anchor. The position correction gain parameter, i.e., the dynamic stiffness coefficient. The dynamic stiffness coefficient determines the weight strength of the system's correction of virtual object positions based on semantic anchors. To achieve cross-device adaptive interaction, the parameter coupling unit establishes a coupling model driven by both visual masking effects and statistical uncertainty.

[0110] The parameter coupling unit receives the viewport texture entropy calculated by the environment perception module 300. and the fusion covariance matrix output by the statistical fusion module 200 The parameter coupling unit first calculates the trace of the fused covariance matrix. This value represents the energy level of the system's current overall positioning error.

[0111] Based on the above input parameters, the parameter coupling unit calculates the dynamic stiffness coefficient according to the following exponential coupling formula. ;

[0112] ;

[0113] In the formula, This represents the dynamic stiffness coefficient, used to adjust the intensity of subsequent potential energy field or pose correction. This represents the system's reference stiffness coefficient, a preset constant, typically determined based on the system's lowest response frequency. This represents the uncertainty gain factor, a positive real number used to adjust the amplification ratio of position uncertainty to the spring constant. Represents the error covariance matrix after fusion. The trace is used to measure overall location uncertainty. This represents the error covariance matrix of the fused state estimation. This represents the texture suppression attenuation factor, a positive real number used to control the rate at which texture complexity suppresses the stiffness coefficient. Represents the texture entropy of the viewport or ROI region. Denotes the base of the natural logarithm. This represents the stiffness coefficient that decreases as texture complexity increases.

[0114] The coupling formula achieves the following technical effects:

[0115] when When the value increases, it indicates a decrease in the reliability of sensor data and an increase in the risk of positioning drift. At this point, the formula... Item makes Increase and enhance the correction weight of semantic anchors, and use environmental geometric features to constrain divergent localization results.

[0116] when When the value increases, it indicates a more complex background texture, reducing the human eye's sensitivity to minute movements of virtual objects. At this point, the formula... Item makes Reducing the correction weight allows virtual objects to retain the sensor's original motion characteristics within a certain range, thus avoiding image jumps caused by overcorrection.

[0117] when When the stiffness coefficient is smaller, it increases to ensure that the virtual object fits tightly against the physical edge to prevent noticeable visual slippage.

[0118] The nonlinear offset correction unit receives the current fusion position from the statistical fusion module 200. Semantic anchors from anchor matching units and the dynamic stiffness coefficient from the parametric coupling unit. To avoid rendering screen oscillations or overshoot caused by simple linear interpolation, the nonlinear offset correction unit uses a damped correction model with saturation constraints to calculate the position offset.

[0119] Nonlinear offset correction unit receives fused position semantic anchors and dynamic stiffness coefficient A damped correction model with saturation constraints is used to calculate the position offset. First, a nonlinear offset correction unit calculates the original deviation vector. Subsequently, the dynamic stiffness coefficient was combined with... With time step Calculate the instantaneous correction increment within a single frame. In this process, a nonlinear saturation function is introduced. To limit the correction magnitude per frame;

[0120] ;

[0121] In the formula, This represents the instantaneous position correction increment in a single frame. This represents the dynamic stiffness coefficient, used to adjust and correct the strength. The initial bias vector represents the geometric offset from the fusion location to the optimal semantic anchor point. This indicates the time step, corresponding to the current frame rendering interval of the system. This represents the preset maximum allowable displacement threshold per frame. This represents a nonlinear saturation function used to limit the magnitude of the correction increment.

[0122] The instantaneous correction increment is calculated. Subsequently, the nonlinear offset correction unit uses a drift accumulation model to maintain the transformation relationship between the rendering coordinate system and the sensor coordinate system, updating the accumulated position offset in each rendering frame. And generate the final virtual camera position used for graphics rendering. .

[0123] For rotational attitude correction, a nonlinear offset correction unit performs attitude compensation based on small-angle assumptions. The rendered attitude is calculated using spherical linear interpolation (Slerp). The interpolation coefficients are determined by the product of the sensitivity coefficient, the dynamic stiffness coefficient, and the time step. Finally, the nonlinear offset correction unit constructs the standard view matrix. Considering the requirements of the graphics rendering pipeline, the view matrix is ​​constructed from the transpose of the rotation matrix corresponding to the rendering pose quaternion and the rendering position.

[0124] See attached document Figure 6 , Figure 6 This is a schematic diagram illustrating the logical principle of the rendering interaction module in this embodiment. The rendering interaction module 500 specifically includes a projection transformation unit, a probabilistic rendering unit, and an interaction determination unit. This module utilizes the fusion state error covariance matrix generated in the aforementioned steps to transfer uncertainty from the three-dimensional physical space to the two-dimensional screen space, thereby achieving adaptive compensation for positioning errors at the visual presentation and user interaction levels.

[0125] The projection transformation unit is configured to perform error propagation calculations from 3D space to 2D image space. In traditional graphics rendering pipelines, typically only the model-view-projection (MVP) transformation is performed on the object's position coordinates. This embodiment goes beyond this, further performing nonlinear propagation of the error distribution. The projection transformation unit receives the position covariance matrix of the virtual object in the camera coordinate system at the current moment. Location covariance matrix From the fusion covariance matrix via view matrix rotational component The transformation yields the result, i.e., the calculation. .

[0126] Since the perspective projection process is nonlinear, the projection transformation unit uses a first-order Taylor expansion approximation, obtained through the Jacobian matrix of the perspective projection function. The 3D covariance matrix is ​​propagated to the 2D screen space to generate the screen space covariance matrix. ;

[0127] ;

[0128] In the formula, Represents the screen space covariance matrix. This represents the covariance matrix of the target point in the camera coordinate system in three-dimensional space. The first-order Jacobian matrix representing the perspective projection function. This represents the transpose of the Jacobian matrix.

[0129] The matrix generated by this calculation Geometrically, an error confidence ellipse is defined on the screen plane. Furthermore, the projection transformation unit extracts... Variance components corresponding to depth direction , as a parameter of depth uncertainty.

[0130] The probabilistic rendering unit is configured to perform probabilistic soft occlusion rendering based on the aforementioned uncertainty parameters to resolve depth conflicts between virtual objects and real occlusions caused by positioning jitter. The probabilistic rendering unit performs calculations at the fragment shader stage of the graphics rendering pipeline. It samples the real-time environmental depth texture generated by the AR device to obtain the real-world scene depth value corresponding to the current pixel. .

[0131] ;

[0132] In the formula, This represents the visibility probability of a virtual fragment relative to the real scene. This represents the depth value of the real scene at the current pixel, and is considered a deterministic mean. This represents the expected depth value of the virtual fragment. This represents the variance of the virtual fragment in the depth direction. The standard deviation represents the depth of uncertainty. The error function is represented by the standard form of the cumulative distribution function of the Gaussian distribution. This represents the Gaussian distribution normalization constant.

[0133] The interaction determination unit is configured to construct an interaction hotspot based on anisotropic errors. The interaction determination unit utilizes the screen space covariance matrix. A statistical extension of the collision bounding box of a virtual object is equivalent to performing a Minkowski sum operation based on the error ellipse on the collider.

[0134] In practice, for user touch points on the screen Projection point of virtual object center The interactive decision unit calculates the Mahalanobis distance between the two. and will The condition for determining a hit is that the value is less than a preset threshold. Geometrically, this determination logic is equivalent to assigning a virtual object an anisotropic interaction region that dynamically deforms with the error, thereby improving the success rate of human-computer interaction in highly dynamic scenarios.

[0135] The device for the cross-device pose collaboration and occlusion perception multi-scenic area AR interaction method described below can be referred to in correspondence with the cross-device pose collaboration and occlusion perception multi-scenic area AR interaction system described above.

[0136] Please see the appendix Figure 7 The present invention also provides a multi-scenic area AR interaction method with cross-device pose collaboration and occlusion perception, including the following steps;

[0137] Configure a heterogeneous data acquisition module to collect sensor data from mobile computing devices and augmented reality display devices, and output the state estimates of each device in the local coordinate system and the corresponding state error covariance matrix;

[0138] The configuration statistical fusion module receives the pose state and state error covariance matrix from multiple heterogeneous sources, performs data fusion calculation using the covariance intersection algorithm, and outputs the fused pose estimate and the fusion covariance matrix representing the confidence interval of the position estimate.

[0139] The environment awareness module is configured to extract the real background image within the viewport and calculate the viewport texture entropy, while simultaneously acquiring geometric edge features in the real environment to generate a semantic edge set.

[0140] The configuration potential field correction module uses the fusion covariance matrix to construct a spatial Mahalanobis distance field, searches for the real edge point that minimizes the Mahalanobis distance in the semantic edge set as the anchor point, and calculates the nonlinear offset vector of the virtual object relative to the real edge to output the corrected pose.

[0141] The configuration rendering interaction module uses the fusion covariance matrix to generate a screen error ellipse and depth variance, calculates the visibility probability based on the depth variance to perform probabilistic soft occlusion rendering, and uses the screen error ellipse to generate an anisotropically expanded interactive hotspot.

[0142] The method in this embodiment can be used to execute the above system embodiment, and its principle and technical effect are similar, so it will not be described again here.

Claims

1. A multi-scene AR interaction system for cross-device pose coordination and occlusion perception, characterized in that, The method comprises the following steps: a heterogeneous data acquisition module configured to acquire sensor data of a mobile computing device and an augmented reality display device, and output state estimation values of each device in a local coordinate system and corresponding state error covariance matrices; a statistical fusion module connected to the heterogeneous data acquisition module, configured to receive multi-source heterogeneous pose states and state error covariance matrices, perform data fusion calculation using a covariance intersection algorithm, and output a fused pose estimation value and a fused covariance matrix representing a position estimation confidence interval; an environment perception module configured to extract real background images within a viewport and calculate viewport texture entropy, and obtain geometric edge features in a real environment to generate a semantic edge set; a potential field correction module connected to the statistical fusion module and the environment perception module, configured to construct a spatial Mahalanobis distance field using the fused covariance matrix, search for a real edge point in the semantic edge set that minimizes the Mahalanobis distance as an anchor point, and calculate a nonlinear offset vector of a virtual object relative to the real edge to output a corrected pose; a rendering interaction module connected to the potential field correction module, configured to generate a screen error ellipse and a depth variance using the fused covariance matrix, calculate a visibility probability based on the depth variance to perform a probabilistic soft occlusion rendering, and generate an anisotropic inflated interaction hot zone using the screen error ellipse; The potential field correction module comprises a potential field construction unit, which is configured to: extract a position error covariance matrix from the fused covariance matrix; construct a scalar potential field based on Mahalanobis distance using the position error covariance matrix, the scalar potential field defining a weighted statistical distance of any point in space relative to the current fused position estimation value; based on the eigenvalue decomposition characteristics of the position error covariance matrix, the scalar potential field adaptively weights the spatial error, constructs a negative correlation mapping relationship between distance gradient and directional variance, assigns the minimum distance gradient to the direction corresponding to the maximum eigenvalue of the position error covariance matrix to reduce the correction constraint, and assigns the maximum distance gradient to the direction corresponding to the minimum eigenvalue of the position error covariance matrix to enhance the position correction constraint.

2. The multi-scene area AR interaction system of cross-device pose collaboration and occlusion perception according to claim 1, characterized in that, The statistical fusion module is specifically configured to: receive the mobile terminal state vector and its covariance matrix output by the heterogeneous data acquisition module, and the wearable terminal state vector and its covariance matrix, interpolate and align the heterogeneous data based on timestamps; convert the data in the local coordinate system of different devices to a unified fusion coordinate system using a hand-eye extrinsic parameter matrix, and perform a similarity transformation based on a rotation matrix for the state error covariance matrix; define an optimization objective function to find optimal weighting coefficients such that the trace of the fused state error covariance matrix is minimized; based on the optimal weighting coefficients, weight and sum the information matrices of the aligned mobile terminal and wearable terminal, and obtain the fused state error covariance matrix by inverting the sum.

3. The multi-scene area AR interaction system of cross-device pose collaboration and occlusion perception according to claim 1, characterized in that, The environment perception module is specifically configured to: Gradient statistical analysis is performed on image data in a current time instant viewport, a normalized gradient amplitude histogram is constructed, and a viewport texture entropy is calculated by using a Shannon entropy formula to represent texture complexity; In communication with the edge computing node, the current fusion position is uploaded and the surrounding lightweight vector map data is pulled; The vector map data is subjected to frustum culling, and the filtered three-dimensional vector features are converted from the geodetic coordinate system to the local fusion coordinate system to generate the semantic edge set.

4. The multi-scene area AR interaction system of cross-device pose collaboration and occlusion perception according to claim 1, characterized in that, The potential field correction module further includes an anchor point matching unit, which is configured to: Iterate through the geometric elements in the semantic edge set to solve an optimization problem to determine an optimal anchor point; the optimization problem is to find a geometric feature point that minimizes the Mahalanobis distance; When the element in the semantic edge set is a line segment, the optimal anchor point is a projection point on the line segment that minimizes the quadratic form value corresponding to the Mahalanobis distance, and the optimal anchor point is located in the direction with the largest system positioning uncertainty.

5. The multi-scene-area AR interaction system of cross-device pose collaboration and occlusion perception according to claim 4, characterized in that, The potential field correction module further includes a parameter coupling unit, which is configured to: Calculate the trace of the fusion covariance matrix to represent the overall positioning error energy; calculate a dynamic stiffness coefficient according to an exponential coupling model, and the dynamic stiffness coefficient determines the weight strength of the system for correcting the virtual object position according to the semantic anchor point; The calculation method of the dynamic stiffness coefficient is: based on the reference stiffness coefficient, multiplied by an enhancement term constructed by the uncertainty gain factor and the trace of the fusion covariance matrix, and multiplied by an exponential decay term constructed by the texture suppression decay factor and the viewport texture entropy.

6. The multi-scene-area AR interaction system of cross-device pose collaboration and occlusion perception according to claim 5, characterized in that, The potential field correction module further includes a nonlinear offset correction unit, which is configured to: Calculate the original deviation vector from the current fusion position to the optimal anchor point; combine the dynamic stiffness coefficient and the time step to calculate the instantaneous correction increment within a single frame; Introduce a nonlinear saturation function to limit the instantaneous correction increment, and perform truncation processing when the input vector length exceeds a preset threshold; Update the cumulative position offset using a drift accumulation model, and superimpose the offset on the fusion position to generate a virtual camera position; Calculate the rendering pose using spherical linear interpolation, and construct a view matrix based on the transpose of the rotation matrix of the rendering pose and the virtual camera position.

7. The multi-scene-area AR interaction system of cross-device pose collaboration and occlusion perception according to claim 1, characterized in that, The rendering interaction module includes a projection transformation unit and a probability rendering unit: The projection transformation unit is configured to use the Jacobian matrix of the perspective projection function to propagate the position covariance matrix in the camera coordinate system to the screen space to generate a screen space covariance matrix, and extract the variance component in the depth direction as a depth uncertainty parameter; The probability rendering unit is configured to sample the environment depth texture to obtain a real scene depth value, and calculate the visibility probability of a virtual pixel with respect to the real scene by using the standard form of the Gaussian distribution cumulative distribution function; The calculation input of the visibility probability includes the difference between the real scene depth value and the virtual pixel expected depth value, and the depth uncertainty parameter.

8. The multi-scene-area AR interaction system of cross-device pose collaboration and occlusion perception according to claim 7, characterized in that, The rendering interaction module further includes an interaction judgment unit, which is configured to: Statistically expanding a collision bounding box of the virtual object using the screen space covariance matrix; calculating a Mahalanobis distance between a user touch point on the screen and a projection point of a center of the virtual object; Taking the Mahalanobis distance being less than a preset threshold as a condition for a hit determination, thereby constructing an interactive hot zone based on anisotropic error.

9. The method of claim 1-8, applied to the multi-scene AR interaction system of claim 1-8, wherein, The method comprises the following steps: Configuring a heterogeneous data acquisition module to acquire sensor data of a mobile computing device and an augmented reality display device, and outputting state estimation values of each device in a local coordinate system and corresponding state error covariance matrices; Configuring a statistical fusion module to receive multi-source heterogeneous pose states and state error covariance matrices, perform data fusion calculation using a covariance intersection algorithm, and output a fused pose estimation value and a fused covariance matrix representing a confidence interval of the position estimation; Configuring an environment perception module to extract real background images in a viewport and calculate viewport texture entropy, and obtain geometric edge features in a real environment to generate a semantic edge set; Configuring a potential field correction module to construct a spatial Mahalanobis distance field using the fused covariance matrix, search for a real edge point in the semantic edge set that minimizes the Mahalanobis distance as an anchor point, and calculate a nonlinear offset vector of the virtual object relative to the real edge to output a corrected pose; Configuring a rendering interaction module to generate a screen error ellipse and a depth variance using the fused covariance matrix, calculate a visibility probability based on the depth variance to perform probabilistic soft occlusion rendering, and generate an anisotropically expanded interactive hot zone using the screen error ellipse.

Images