A Mixed Reality Spatial Localization Method Based on Multi-Source Image Fusion

Abstract

This invention relates to the field of computer vision positioning technology. Specifically, it relates to a mixed reality spatial positioning method based on multi-source image fusion. The invention first simultaneously acquires RGB images, depth images, and original polarized images of a scene, generates a polarization angle map, and performs multimodal image alignment. Corner detection and depth curvature verification are used to identify weakly textured plane regions, and depth gradients are used to segment their boundaries. Based on the Fresnel reflection model, the surface normal azimuth angle of the weakly textured plane is inversely solved from the polarization angle map and converted into a unit normal vector. A one-dimensional virtual constraint edge is constructed that only constrains normal translation. Finally, this virtual constraint edge, along with visual reprojection error constraint edges and depth point cloud ICP registration error constraint edges, is added to the pose graph optimization framework. The Levenberg-Marquardt algorithm is used to jointly solve the problem, outputting a corrected camera pose after suppressing normal drift, thus improving the robustness and accuracy of mixed reality spatial positioning.

Description

Technical Field

[0001] This invention relates to the field of computer vision positioning technology, and more specifically, to a mixed reality spatial positioning method based on multi-source image fusion. Background Technology

[0002] In the development of mixed reality and spatial positioning technology, existing research has explored multiple levels. Chinese patent application CN118776565A discloses an RGB-D inertial pose estimation method, system and device based on multi-dimensional geometric features. By extracting point and line features and establishing vanishing point measurement residuals, it can maintain positioning accuracy in weak texture degradation environments. Chinese patent application CN117647217A discloses a mixed reality device orientation and object positioning method, which aims to achieve accurate orientation of head-mounted devices through multi-source sensor fusion. In specialized research on weak texture scenes, academia and industry have also proposed methods such as fusing point and line features, introducing semantic planes and using the Manhattan world hypothesis to make up for the shortcomings of traditional point feature extraction that cannot be stably extracted.

[0003] However, most of the above technologies still rely on extracting repeatable texture signals such as corners, edges or lines from two-dimensional images. In the weak texture plane areas of real mixed reality scenes such as solid color walls, smooth floors, and matte countertops, even if line features and depth information are fused, the lack of stable local gradient field distribution results in low feature descriptor discrimination and frequent mismatches, ultimately leading to tracking loss or continuous pose drift in the normal translation direction.

[0004] Meanwhile, polarization imaging and its physical advantages in normal estimation have attracted industry attention. Chinese patent application CN119478254A discloses a 3D reconstruction method and system based on polarization fringe projection structured light fusion, which uses four images with different polarization angles to solve the azimuth and zenith angles of the surface normal and uses them for high-precision 3D reconstruction. However, these studies are mostly focused on the geometric reconstruction of offline or static scenes and have not yet organically embedded real-time polarization angle information into the pose graph optimization loop of mixed reality. Furthermore, there is a lack of lightweight modules that can be integrated into the existing SLAM framework, such as one-dimensional virtual constraint edges that only constrain normal translation for weakly textured planes. Therefore, how to simultaneously utilize the physical constraint of polarization angle on the normal of weakly textured planes during real-time localization and mapping, construct virtual constraints that specifically suppress normal drift, and jointly optimize them with visual reprojection error and depth point cloud registration error has become a problem for mixed reality head-mounted devices to achieve long-range stable and high-precision localization in weakly textured scenes. To solve this technical problem, we provide a mixed reality spatial localization method based on multi-source image fusion. Summary of the Invention

[0005] The purpose of this invention is to provide a mixed reality spatial positioning method based on multi-source image fusion to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, one objective of this invention is to provide a mixed reality spatial localization method based on multi-source image fusion, comprising the following steps:

[0007] S1. Synchronously acquire RGB images, depth images and polarization original images of the scene through a mixed reality head-mounted device, calculate and generate a polarization angle map from the polarization original image, and perform time stamp alignment and geometric calibration on the RGB images and depth images to establish a pixel-by-pixel mapping relationship.

[0008] S2. The corner detection algorithm is used to extract feature points from the RGB image. Continuous image regions with feature point response values ​​lower than a preset threshold and point cloud curvature less than the curvature threshold after depth image verification are marked as weak texture plane regions. The boundaries of the weak texture plane regions are segmented based on the depth gradient information of the depth image.

[0009] S3. For the weak texture plane region, extract the corresponding local polarization angle block from the polarization angle map. Based on the Fresnel reflection model, use the single-value mapping relationship between the polarization angle of the reflected light and the azimuth angle of the surface normal when the incident angle is less than the Brewster angle. Solve the azimuth angle component of the surface normal vector of the weak texture plane region in the camera coordinate system from the local polarization angle block.

[0010] S4. Convert the azimuth component of the surface normal vector into a unit normal vector in the camera coordinate system, and construct a one-dimensional virtual constraint edge acting on the current camera pose. The one-dimensional virtual constraint edge is configured to use only the translation deviation between the current frame and the previous keyframe along the direction of the unit normal vector as the residual term. Its information matrix is ​​assigned a preset Gaussian weight in the normal direction and a value of zero in the tangential direction.

[0011] S5. Add one-dimensional virtual constraint edges as new constraint terms, and combine them with visual reprojection error constraint edges and depth point cloud ICP registration error constraint edges into the pose graph optimization framework. Use the Levenberg-Marquardt algorithm for joint nonlinear least squares optimization to solve the problem and output the corrected camera pose after unit normal vector direction drift suppression.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] This invention generates a polarization angle map by simultaneously acquiring RGB images, depth images, and polarized raw images. It then identifies weakly textured plane regions from the polarization angle map and uses a Fresnel reflection model to inversely solve for the surface normal azimuth angle of the region. This allows for the construction of a one-dimensional virtual constraint edge that only constrains the normal translation. This constraint edge is then combined with traditional visual reprojection errors and depth point cloud ICP registration errors into a pose graph optimization framework for joint solution. This solves the problem of continuous camera normal drift caused by the lack of sufficient visual features in weakly textured scenes such as solid-color walls and smooth floors. Without introducing excessive computational burden, it suppresses the normal pose drift of weakly textured planes, improving the positioning robustness and long-term stability of mixed reality headsets in complex indoor environments. Attached Figure Description

[0014] Figure 1 This is an overall working diagram of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1 As shown, this embodiment provides a mixed reality spatial localization method based on multi-source image fusion, including the following steps:

[0017] S1. Synchronously acquire RGB images, depth images and polarization original images of the scene through a mixed reality head-mounted device, calculate and generate a polarization angle map from the polarization original image, and perform time stamp alignment and geometric calibration on the RGB images and depth images to establish a pixel-by-pixel mapping relationship.

[0018] S2. The corner detection algorithm is used to extract feature points from the RGB image. Continuous image regions with feature point response values ​​lower than a preset threshold and point cloud curvature less than the curvature threshold after depth image verification are marked as weak texture plane regions. The boundaries of the weak texture plane regions are segmented based on the depth gradient information of the depth image.

[0019] S3. For the weak texture plane region, extract the corresponding local polarization angle block from the polarization angle map. Based on the Fresnel reflection model, use the single-value mapping relationship between the polarization angle of the reflected light and the azimuth angle of the surface normal when the incident angle is less than the Brewster angle. Solve the azimuth angle component of the surface normal vector of the weak texture plane region in the camera coordinate system from the local polarization angle block.

[0020] S4. Convert the azimuth component of the surface normal vector into a unit normal vector in the camera coordinate system, and construct a one-dimensional virtual constraint edge that acts on the current camera pose. The one-dimensional virtual constraint edge is configured to use only the translation deviation between the current frame and the previous keyframe along the unit normal vector direction as the residual term. Its information matrix is ​​assigned a preset Gaussian weight in the normal direction and a value of zero in the tangential direction.

[0021] S5. Add one-dimensional virtual constraint edges as new constraint terms, and combine them with visual reprojection error constraint edges and depth point cloud ICP registration error constraint edges into the pose graph optimization framework. Use the Levenberg-Marquardt algorithm for joint nonlinear least squares optimization to solve the problem and output the corrected camera pose after unit normal vector direction drift suppression.

[0022] After completing multi-sensor collaborative initialization, the mixed reality headset initiates polarization optical feature extraction and polarization angle map construction. This is achieved through a standardized execution method that simultaneously acquires multi-directional polarized raw images, calculates Stokes parameters pixel-by-pixel, analyzes the polarization angle of reflected light, and completes multi-image spatiotemporal alignment. This transforms the polarization physical features of the scene surface into quantifiable image data, effectively supplementing the visual feature information of weakly textured areas and providing optical basis for subsequent plane normal inverse kinematics. The mixed reality headset integrates a dedicated polarization camera in its front-end sensing module. The polarization camera is an image acquisition device with a built-in micro-polarization array filter, consisting of zero-degree and forty-five-degree filters. The device consists of periodically arranged micro-polarizer units with four standard polarization directions: 90 degrees, 135 degrees, etc. This allows for the simultaneous acquisition of original polarized images of the same scene in multiple different polarization directions at a single imaging moment. These original polarized images are two-dimensional image data recording the distribution of reflected light intensity in a scene under a specific polarization direction. The mixed reality head-mounted device uses hardware-level synchronous trigger signals to control the RGB image sensor, depth image sensor, and polarization camera to complete scene image acquisition at the same time point, ensuring the temporal consistency of the acquired RGB image, depth image, and original polarized images in multiple polarization directions. After acquisition, each original polarized image is processed pixel-by-pixel grayscale. Grayscale value extraction: Grayscale value is a digital numerical value characterizing the light intensity received by a single pixel, directly reflecting the intensity of reflected light in the corresponding polarization direction. Then, based on the grayscale values ​​corresponding to the four standard polarization directions, the Stokes parameters for each pixel are calculated according to the standard optical measurement formula. Stokes parameters are a set of physical parameters that completely describe the polarization state of light waves, including total light intensity parameters, linear polarization horizontal and vertical component parameters, linear polarization 45-degree and 135-degree component parameters, and circular polarization component parameters. The total light intensity parameter is calculated by adding the grayscale value in the 0-degree polarization direction and the grayscale value in the 90-degree polarization direction, and then dividing by two. The linear polarization horizontal and vertical component parameters... The calculation method is to subtract the gray value of the 90-degree polarization direction from the gray value of the zero-degree polarization direction. The calculation method for the linearly polarized 45-degree and 135-degree component parameters is to subtract the gray value of the 135-degree polarization direction from the gray value of the 45-degree polarization direction. The circularly polarized component parameters are not involved in the calculation of the polarization angle in this calculation; they are only recorded numerically. After calculating the Stokes parameters for all pixels, the polarization angle of the reflected light at each pixel is calculated using the total light intensity parameter and the linearly polarized horizontal and vertical component parameters. The polarization angle is an angular physical quantity describing the polarization direction of the reflected light and has a fixed mapping relationship with the normal azimuth angle of the object surface. The specific calculation method for the polarization angle is as follows:

[0023] The polarization angle is equal to half the result of the arctangent function. The input variable of the arctangent function is the horizontal and vertical linear polarization components divided by the total light intensity. The initial unit of the calculated polarization angle is radians, which need to be uniformly converted to angle values ​​for subsequent data processing and image encoding. After the polarization angle values ​​of all pixels are calculated, the polarization angle map generation and multi-image alignment stage begins. The polarization angle map is a two-dimensional image that presents the polarization angle value corresponding to each pixel in a fixed grayscale gradient encoding form, maintaining the same image resolution and field of view as the RGB image and depth image. First, a timestamp alignment operation is performed to extract the acquisition timestamp values ​​of the RGB image, depth image, and polarization angle map. Images with a timestamp difference of less than one millisecond among the three sets of data are considered valid synchronization data. Abnormal image frames with mismatched timestamps are removed to ensure temporal consistency among the three sets of images. Subsequently, a geometric calibration operation is performed, calling the camera calibration data completed before the mixed reality headset left the factory. This camera calibration data includes camera intrinsic and extrinsic parameters. The intrinsic parameters include the image sensor focal length, principal point coordinates, and lens distortion coefficients. The extrinsic parameters include the relative positions and attitude transformation relationships between the RGB image sensor, depth image sensor, and polarization camera. The pixel coordinates of the polarization angle map are uniformly converted to the pixel coordinates of the RGB and depth images using the camera calibration data, establishing the temporal consistency among the three sets of images. Pixel-by-pixel mapping means that any pixel coordinate position in the polarization angle map corresponds to the exact same pixel coordinate position in the RGB image and the depth image. The pixels in the three sets of images achieve a one-to-one correspondence without offset or misalignment. This ultimately generates a standardized polarization angle map that is timestamped and geometrically calibrated with the RGB and depth images. For example, a mixed reality head-mounted device might acquire an RGB image, a depth image, and four polarization directions with a resolution of 1920 x 1080. A pixel might have a grayscale value of 126 in the 0-degree polarization direction, 118 in the 45-degree polarization direction, and 124 and 135 in the 90-degree polarization direction. With a grayscale value of 120 in the polarization direction, the calculated total light intensity parameter is 125, the linear polarization horizontal and vertical component parameter is 2, and the linear polarization 45-degree and 135-degree component parameter is -2. Substituting these values ​​into the formula, the polarization angle is calculated as 1 / 2 multiplied by the arctangent 2 divided by 125, which is approximately 0.45 degrees. This value is used as the polarization angle value of the corresponding pixel and filled into the polarization angle map. Then, coordinate transformation and pixel-by-pixel mapping are completed through camera intrinsic and extrinsic parameters to ensure that the polarization angle map is completely aligned with the depth image space of the RGB image, thus ensuring the accuracy and stability of the polarization angle map. This provides a reliable data foundation for subsequent local polarization corner block truncation in weak textured plane areas and inverse solution of surface normal vector azimuth angle.

[0024] After the polarization angle map is aligned with the timestamps and geometrically calibrated with the RGB and depth images, spatial consistency filtering is immediately initiated. Spatial consistency filtering is a pixel-level noise correction operation performed based on the spatial continuity of the polarization angles of an object's surface. It effectively removes random noise and outliers generated during polarization angle calculation, improving the spatial distribution stability of the polarization angle map and providing high-precision basic data for subsequent local polarization angle patch extraction and surface normal inverse kinematics. The execution process of spatial consistency filtering begins by traversing each pixel to be processed in the polarization angle map, starting with:

[0025] The current pixel defines a local neighborhood, which is a square set of pixels extending outwards from the current pixel. This local neighborhood forms the basis for calculating the local polarization angle statistical features. Initially, the local neighborhood uses a fixed size of 3x3. Then, the polarization angle statistical features of all pixels within the corresponding local neighborhood are calculated. These polarization angle statistical features are a set of quantitative indicators describing the distribution of polarization angle values ​​within the local neighborhood. They include the arithmetic mean, variance, mode, and polarization angle consistency measure of all pixels within the neighborhood. The arithmetic mean is calculated by summing the polarization angle values ​​of all pixels within the local neighborhood and dividing by the total number of pixels in the neighborhood. The variance is calculated by adding the polarization angle values ​​of each pixel within the local neighborhood. The difference between the value and the arithmetic mean is squared, all squared values ​​are summed, and then divided by the total number of pixels in the neighborhood. The mode is the polarization angle value with the highest frequency in the local neighborhood, and it is also the benchmark value for subsequent replacement of abnormal pixels. The polarization angle consistency metric is a specific indicator used to quantify the concentration of polarization angle distribution in a local neighborhood. It is calculated by subtracting the local neighborhood polarization angle variance from the global maximum variance of the polarization angle map. The global maximum variance is the maximum value among all local neighborhood variance values ​​in the current polarization angle map. The polarization angle consistency metric ranges from zero to one. A larger value indicates a more concentrated polarization angle distribution and better spatial continuity in the neighborhood, while a smaller value indicates a more dispersed polarization angle distribution and poorer spatial continuity. After calculating the statistical characteristics of the polarization angle, the process enters the adaptive adjustment stage of the spatial consistency filtering parameters. This stage is an innovative technical feature of spatial consistency filtering, which dynamically matches the processing parameters according to the polarization angle distribution characteristics of different regions. First, the size of the local neighborhood is adaptively adjusted based on the polarization angle consistency metric. The adjustment rule is as follows: when the polarization angle consistency metric is greater than or equal to 0.9, the current region is determined to be a planar region with a highly concentrated polarization angle distribution, and the local neighborhood size is maintained at 3x3. When the polarization angle consistency metric is between 0.7 and 0.9, the current region is determined to be a transitional region with a moderately concentrated polarization angle distribution, and the local neighborhood size is adjusted to 5x5. When the polarization angle consistency metric is less than 0.7, the current region is determined to be a region with a relatively concentrated polarization angle distribution. For scattered textures or edge regions, the local neighborhood size is adjusted to 7x7. After the neighborhood size adjustment, the difference judgment threshold is adaptively adjusted based on the polarization angle consistency metric. The difference judgment threshold is a critical difference used to determine whether the current pixel's polarization angle value is an outlier. The adjustment rule is as follows: when the polarization angle consistency metric is greater than or equal to 0.9, the difference judgment threshold is set to 2 degrees; when the polarization angle consistency metric is between 0.7 and 0.9, the difference judgment threshold is set to 3 degrees; and when the polarization angle consistency metric is less than 0.7, the difference judgment threshold is set to 5 degrees. After completing the adaptive adjustment of the neighborhood size and the difference judgment threshold, the absolute difference between the current pixel's polarization angle value and the arithmetic mean of the local neighborhood polarization angles is calculated.The absolute difference is calculated by subtracting the arithmetic mean of the neighborhood from the current pixel's polarization angle value and taking the non-negative value. This calculated absolute difference is then compared to an adaptively adjusted difference threshold. If the absolute difference is greater than the threshold, the current pixel's polarization angle is determined to have a difference exceeding a preset standard, classifying it as an abnormal noise pixel. In this case, the original polarization angle value of the current pixel is discarded and replaced with the mode value represented by the local neighborhood's polarization angle statistical features. If the absolute difference is less than or equal to the threshold, the current pixel's polarization angle value is considered valid, and the original value is retained without replacement. This process is repeated for each pixel in the polarization angle map, completing noise assessment and value replacement for all pixels. Finally, a standardized polarization angle map, processed by spatial consistency filtering, is obtained, for example, for a weak texture plane. Within the region, the local neighborhood polarization angle consistency metric is 0.95, the neighborhood size remains 3x3, the difference threshold is 2 degrees, the current pixel polarization angle value is 15 degrees, the neighborhood arithmetic mean is 5 degrees, the absolute difference is 10 degrees (a value greater than 2 degrees is considered an abnormal pixel), and the neighborhood mode is 6 degrees (the current pixel polarization angle is replaced with 6 degrees). For edge region pixels, the polarization angle consistency metric is 0.6, the neighborhood size is adjusted to 7x7, the difference threshold is 5 degrees, and the difference between the current pixel polarization angle value and the neighborhood mean is 4 degrees (a value less than the threshold is retained). All adjustments are automatically performed based on the polarization angle consistency metric. Noise removal and value replacement follow the spatial continuity of polarization angles in planar regions, preserving the true polarization angle characteristics to the greatest extent while eliminating abnormal noise, providing reliable data support for subsequent polarization angle analysis and inverse normal derivation in weak-texture planar regions.

[0026] After noise removal and numerical optimization of the polarization angle map following spatial consistency filtering, mixed reality spatial localization initiates the identification and labeling of weakly textured plane regions. This dual-judgment logic, which integrates 2D visual feature detection and 3D geometric curvature verification, solves the problem that a single visual feature cannot distinguish between weakly textured planes and weakly textured surfaces. It identifies regions with no obvious visual features but flat surfaces, providing a stable region carrier for subsequent polarization angle inverse normal determination, effectively improving the robustness of mixed reality spatial localization in weakly textured scenes. First, corner detection is performed on the RGB images acquired and spatiotemporally aligned by the mixed reality head-mounted device. Corner detection is a feature extraction operation in computer vision that extracts local gray-level abrupt changes or edge extrema points in an image. In this study, Haar is used. The Harris corner detection algorithm performs full-frame feature point extraction. It identifies spatially stable corners by calculating the sum of squares of grayscale changes in multiple directions (horizontal, vertical, etc.) within a local sliding window of the image, defining these as feature points. Feature points are reference pixels in the RGB image used for visual reprojection and pose calculation. After extracting all feature points, the algorithm calculates the response value for each. The response value is a quantitative indicator output by the Harris corner detection algorithm, used to characterize the saliency and tracking stability of the feature point. The response value is calculated based on the eigenvalues ​​of the local autocorrelation matrix of the image; a larger response value indicates higher visual saliency of the feature point, while a smaller response value indicates a weak feature point. Even pixels lacking features are pre-defined by setting a response value threshold through standard weak texture scene calibration experiments. This threshold serves as a fixed critical value to distinguish between strong and weak feature points. After calculating the response values, all feature points in the RGB image with response values ​​below the preset threshold are selected. These feature points correspond to weak feature pixel locations in the image without obvious texture. Since the timestamp alignment, geometric calibration, and pixel-by-pixel mapping relationship between the RGB image and the depth image have been completed in the early stages, the pixel coordinates of any two-dimensional pixel in the RGB image can be directly converted to the two-dimensional pixel coordinates of the same physical location in the depth image through camera intrinsic and extrinsic parameters. Therefore, the pixel coordinates of the selected weak feature points in the RGB image are mapped one by one to the pixel coordinates in the depth image. The corresponding pixel coordinates are used in the depth image, which is a two-dimensional image captured by the depth sensor of the mixed reality headset. Each pixel location stores the straight-line distance from the corresponding physical point to the camera's optical center, i.e., the depth value, in millimeters. This data is used to construct the 3D point cloud. Based on the mapped depth image coordinates, the center depth value corresponding to that coordinate is extracted, along with eight adjacent depth values ​​within an eight-neighbor range, forming a nine-pixel local depth value set. Based on the camera intrinsic parameter matrix, the two-dimensional pixel coordinates corresponding to the local depth value set are converted into 3D point cloud coordinates in the camera coordinate system. The 3D point cloud is a discrete three-dimensional coordinate set describing the geometry of the scene surface. Each point contains XYZ axis coordinate values ​​and can characterize the spatial shape of the object's surface.Subsequently, the curvature of the 3D point cloud corresponding to the feature point location within a local range is calculated. Curvature is a geometric parameter that quantifies the degree of bending of the local surface of the 3D point cloud. The smaller the curvature, the smoother the surface; the larger the curvature, the more curved the surface. The curvature calculation adopts the least squares local plane fitting method. First, a plane fitting is performed on the local 3D point cloud to obtain the equation of the fitted plane. Then, the sum of squared distances from each 3D point to the fitted plane is calculated. The sum of squared distances is divided by the total number of local point clouds to obtain the final curvature value. A curvature threshold is pre-set through calibration experiments of standard planes and standard curved surfaces. This threshold serves as the geometric critical value for distinguishing between planes and non-planes, thus completing the curvature calculation. After calculation, a pixel location filtering and culling operation is performed. The culling rules are as follows: retain pixels whose feature point response values ​​are below a preset threshold and whose corresponding point cloud curvature is less than a curvature threshold; and remove pixels whose feature point response values ​​are below a preset threshold but whose corresponding point cloud curvature is not less than a curvature threshold. The culled pixel locations correspond to weakly textured but non-planar regions, such as wall edges, object corners, and curved shells. The retained pixel locations correspond to weakly textured and geometrically flat regions. After the culling operation is completed, all pixel locations in the depth image are traversed, and the non-culled pixel locations are identified and their spatial connectivity is determined. Spatial connectivity refers to the existence of four neighbors between pixels in a two-dimensional image. The adjacency connectivity of the domain or eight-neighbor domain is used to determine whether a pixel is connected. Pixels that satisfy the adjacency connectivity are considered connected pixels. Finally, the set of spatially connected pixels that are not eliminated and are surrounded by the remaining eliminated pixel positions in the depth image are uniformly marked as weak texture plane regions. Weak texture plane regions are physical areas in RGB images that have no obvious visual texture but have a flat surface, such as solid-color walls, smooth desktops, matte floors, and other areas where traditional visual localization is prone to failure. For example, in a pure white wall area without any pattern or texture, the feature point response values ​​obtained by corner detection in the RGB image are all below a preset threshold, and the local point cloud curvature calculated after mapping the depth image is all less than a certain value. Curvature thresholding is used to identify areas where pixels are not removed and the overall space is connected, thus marking these areas as weakly textured planar regions. However, at the corner where the wall and cabinet meet, feature point response values ​​are below the threshold, but the point cloud curvature exceeds the threshold; these pixels are removed and not included in the weakly textured planar regions. This full-process marking, combining 2D visual feature saliency and 3D geometric flatness assessment, overcomes the limitations of traditional weakly textured recognition that relies solely on visual features. It filters out planar regions suitable for polarization normal inverse solving, ensuring a solid regional foundation for subsequent surface normal vector solving based on polarization angles and the construction of one-dimensional virtual constraint edges. This fundamentally improves the accuracy and stability of mixed reality spatial positioning.

[0027] After the weak-textured planar regions are marked based on both visual features and 3D curvature, a refined segmentation of the region boundaries based on depth gradient information is immediately initiated. This process employs a multi-level execution approach, including depth gradient abrupt change recognition, omnidirectional ray scanning boundary localization, curve fitting closed contours, and depth calibration optimization, to define the spatial range of the weak-textured planar regions. This addresses the issues of blurred edges and insufficient boundary extraction accuracy in weak-textured regions, providing regional constraints for subsequent local polarization corner block truncation. It also prevents non-planar pixels at the edges from interfering with the surface normal vector inverse solution results, improving the robustness of mixed reality spatial localization. First, for each independently marked weak-textured planar region, a pixel-by-pixel calculation of the depth gradient magnitude is performed. The depth gradient magnitude is the value corresponding to adjacent pixels in the quantized depth image. The depth gradient is a geometric indicator of the drastic change in depth values. The depth value is the linear distance from a physical point stored in a single pixel in a depth image to the camera's optical center. The depth gradient magnitude is calculated using the Sobel gradient operator, which consists of two sets of 3x3 convolution kernels in the horizontal and vertical directions. The horizontal convolution kernel is convolved with a local depth image patch in a weakly textured plane region to obtain the horizontal depth gradient component, and the vertical convolution kernel is convolved with the same local depth image patch to obtain the vertical depth gradient component. The specific calculation method for the depth gradient magnitude is to add the squares of the horizontal and vertical depth gradient components and then take the square root. This value directly reflects the spatial rate of change of depth information. The interior of a weakly textured plane region has a flat surface. The depth gradient magnitude generally remains in a low range. At the boundary between the region edge and other objects or the spatial background, the depth value changes drastically, and the depth gradient magnitude jumps significantly. Through multiple standard planar scene calibration experiments, a gradient threshold is pre-set. This threshold is a fixed critical value that distinguishes pixels within a weakly textured planar region from boundary pixels. If the depth gradient magnitude exceeds this threshold, the pixel position is determined to have entered a non-planar boundary region. After calculating the depth gradient magnitude of all pixels, the geometric center pixel within the current weakly textured planar region is selected as the starting center point for ray scanning. The geometric center pixel is determined by calculating the arithmetic mean of the coordinates of all pixels within the region, ensuring uniform scanning coverage. Subsequently, using the starting center point as the origin, 15... The system divides the area into twenty-four omnidirectional scanning directions at fixed angular intervals, covering the complete angular range from 0 to 360 degrees. For each scanning direction, a ray scan is performed pixel-by-pixel along a straight line, starting from the initial center point. The ray scan path is a continuous sequence of pixel coordinates. During the scan, the depth gradient magnitude of each pixel on the path is read in real time and compared with a preset gradient threshold. When the depth gradient magnitude of a pixel on the scanning path exceeds the gradient threshold for the first time, the scan in that direction is immediately terminated, and the two-dimensional coordinates of that pixel are recorded as candidate boundary points for the current direction. Candidate boundary points are discrete feature points that characterize the edge positions of weakly textured planar regions and are the basic nodes for constructing closed boundaries. The same rules are used to complete the ray scan in all twenty-four directions.Candidate boundary points from all directions are collected to form a complete set of candidate boundary points. After the candidate boundary point set is collected, a quadratic polynomial curve fitting method is used to connect the discrete candidate boundary points continuously. Quadratic polynomial curve fitting is a mathematical method that minimizes the sum of squared distances from the fitted curve to all candidate boundary points and solves the polynomial coefficients to generate a continuous and smooth curve. This method can eliminate positional jumps and random deviations of discrete candidate boundary points. After fitting, all candidate boundary points are connected in an orderly manner to form a preliminary closed boundary surrounding the weak texture plane region. The preliminary closed boundary is a continuous curve contour, but it has local sharp spikes and slight positional deviations, which require further optimization. First, Gaussian smoothing with a 5x5 Gaussian convolution kernel is performed on the preliminary closed boundary. The coordinates of each node on the boundary are weighted and averaged to remove irregular fluctuations and spikes on the boundary, resulting in a smoothed boundary contour. Then, the position of the smoothed boundary nodes is adjusted using depth values. The adjustment process first extracts the depth values ​​of all pixels inside the weak texture plane region and calculates the arithmetic mean depth value, and then extracts the depth values ​​of each pixel. The depth value corresponding to the boundary node is calculated, and the difference between the boundary node depth value and the average depth value of the region is calculated. A preset depth deviation threshold is set. If the difference exceeds the threshold, the boundary node coordinates are shifted inward into the weak texture plane region until the depth difference falls within the deviation threshold range. Finally, a final boundary that perfectly matches the actual physical range of the weak texture plane region is obtained. The final boundary completely delineates the effective pixel set of the weak texture plane region, ensuring that the local polarization angle blocks extracted from the polarization angle map only contain effective pixels of the plane region, and completely eliminating noise data from non-planar areas at the edges. For example, in a weak texture plane region formed by a solid-color matte desktop, the internal depth gradient magnitude is lower than the gradient threshold. Twenty-four candidate boundary points are obtained by scanning the geometric center. After curve fitting, a preliminary closed boundary is formed. After Gaussian smoothing and depth value calibration, a boundary matching the actual contour of the desktop is finally obtained. The entire boundary segmentation relies on the physical abrupt change characteristics of the depth gradient, providing a region carrier for the subsequent inverse solution of the surface normal vector azimuth angle based on the Fresnel reflection model, thus solidifying the algorithmic foundation for mixed reality spatial positioning in weak texture scenes.

[0028] After the final boundary segmentation and effective pixel delineation of the weakly textured planar region are completed, the inverse solution of the normal components based on the mapping relationship between the polarization angle and the surface normal azimuth angle is initiated. This method transforms the polarization angle optical characteristics of the weakly textured planar region into three-dimensional geometric normal parameters through offline calibration to construct a standardized lookup table, online lookup table matching of candidate values, and global statistical selection of the optimal solution. This overcomes the technical bottleneck of being unable to calculate the planar normal in weakly textured scenes due to the lack of visual features, and provides a geometric constraint benchmark for the subsequent construction of one-dimensional virtual constraint edges. First, offline calibration is performed to construct a mapping relationship lookup table. The mapping relationship lookup table is a structured data table that stores the one-to-one correspondence between the polarization angle and the surface normal azimuth angle of the weakly textured planar material under the constraint of the Fresnel reflection model. The Fresnel reflection model describes... This is an optical model that describes the physical relationship between the polarization characteristics of reflected light and the incident angle and surface normal when light is reflected at a medium interface. Brewster's angle is a specific incident angle at which the reflected light is completely linearly polarized. This calibration is limited to the condition that the incident angle is smaller than Brewster's angle. Under this condition, there is a single-valued mapping relationship between the polarization angle of the reflected light and the azimuth angle of the surface normal. The azimuth angle of the surface normal vector is the angle between the projection of the surface normal vector onto the XOY plane of the camera coordinate system and the positive X-axis direction. It is a geometric parameter characterizing the orientation of the plane. The calibration process uses a standard planar calibration plate. The calibration plate is a flat, matte, low-texture plane with uniform material, fixed on a high-precision rotational displacement platform. The azimuth angle of the surface normal vector of the calibration plate is adjusted by rotating the platform, covering the area in 10-degree increments. Within the full angular range of 0 to 360 degrees, the calibration plate's original polarization image is acquired using a mixed reality head-mounted device at each angular step. The corresponding polarization angle value is calculated, and the current precise surface normal azimuth angle value is recorded simultaneously. Each set of polarization angle values ​​is bound and stored with its corresponding surface normal azimuth angle value. The calibration experiment is repeated until all common weakly textured planar materials are covered, including matte plastic, solid color walls, wood panels, and metallic paint. Finally, a complete mapping relationship lookup table is formed, containing polarization angle input fields, surface normal azimuth angle output fields, and material type fields. The lookup table supports index matching of polarization angle values, enabling fast table lookup output. After the offline construction of the mapping relationship lookup table is completed, the online reverse engineering operation stage begins. First, based on the weak... The final boundary of the textured planar region is determined by extracting local polarization angle patches from the spatially consistent polarization angle image. These patches are sub-images containing only valid pixels from the weakly textured planar region; non-planar pixels outside the boundary are completely removed to ensure data purity. Then, each valid pixel within the local polarization patch is traversed, and the corresponding polarization angle value is extracted. This polarization angle value is a standardized angle value calculated using Stokes parameters and corrected by spatial consistency filtering. Using the polarization angle value of each pixel as the search keyword, a matching query is performed in a pre-built mapping lookup table. The query rule is to find the polarization angle record in the table with the smallest difference from the current polarization angle value, and then extract the corresponding surface normal azimuth angle value.This value is used as a candidate value for the azimuth angle of the surface normal vector corresponding to the current pixel. A lookup operation is then performed on all valid pixels within the local polarization block to obtain a set of candidate values ​​for the azimuth angle of the surface normal vector, matching the number of pixels. A global statistical analysis is then performed on this set, using a frequency histogram method. The numerical range of the candidate values ​​is divided into continuous statistical intervals with a step size of 0.5 degrees. The frequency of each candidate value within each interval is recorded. After the statistical analysis, the interval with the highest frequency is selected, and the center value within that interval is extracted as the optimal candidate value. This optimal candidate value is the azimuth component of the surface normal vector of the weak texture plane region in the camera coordinate system. Frequency statistics are used to select the most frequent value. This numerical approach effectively eliminates the effects of local pixel noise, calibration errors, and lookup table biases, ensuring the overall representativeness and geometric accuracy of the azimuth component. For example, a local polarization block contains 500 effective pixels, and a lookup table yields 500 azimuth candidate values. Among these, the candidate value of 45 degrees appears 320 times, a frequency far higher than other values. Ultimately, 45 degrees is selected as the azimuth component of the surface normal vector for this weakly textured plane region. Following the laws of optical physics, the two-dimensional polarization angle feature is transformed into a three-dimensional normal geometric parameter. This provides geometric data for subsequently converting the azimuth component into a unit normal vector in the camera coordinate system and constructing a one-dimensional virtual constraint edge. This effectively suppresses pose drift in mixed reality spatial positioning within the weakly textured plane region, improving the stability and accuracy of positioning.

[0029] After the local polarization corner blocks are cropped, pixel validity correction based on 3D geometry and optical constraints is initiated. This is achieved by back-projecting depth data into 3D coordinates, roughly estimating the normal vector through plane fitting, and verifying the applicability of the model by checking the incident angle. Abnormal pixels in the local polarization corner blocks that do not meet the Fresnel reflection model constraints are eliminated, ensuring that the polarization angle values ​​used in subsequent table lookups and statistics all conform to the physical model's applicability conditions. This improves the accuracy and reliability of the surface normal azimuth inverse solution from the source, providing data validity assurance for weak texture constraints in mixed reality spatial positioning. The depth image is a 2D image acquired by the depth sensor of the mixed reality head-mounted device. Each pixel stores the straight-line distance from the corresponding physical point to the camera's optical center, i.e., the depth value. The depth value is a millimeter-level quantized value. The parameters are the inherent parameter matrix calibrated at the camera's factory, including the image sensor focal length, principal point coordinates, and lens distortion correction coefficients. These parameters convert two-dimensional pixel coordinates into three-dimensional spatial coordinates. The camera coordinate system is a three-dimensional Cartesian coordinate system with the camera's optical center as the origin, the optical axis as the Z-axis, the horizontal to the right as the X-axis, and the vertical upward as the Y-axis. It serves as the reference coordinate system for spatial positioning and geometric calculations. The preliminary three-dimensional coordinates are the original three-dimensional spatial positions of the physical points corresponding to the pixels in the camera coordinate system, calculated by projecting the two-dimensional pixel coordinates and depth values. The initial plane is an approximate plane obtained by fitting the preliminary three-dimensional coordinates of all pixels in a weakly textured plane region. It is used to quickly estimate the coarse normal vector of the plane. The coarse normal vector is the unit normal vector of the initial plane, representing the approximate shape of the plane. Spatial orientation, the angle of incidence of light is the angle between the imaging ray emitted by the camera and the plane normal vector, the Fresnel reflection model is an optical model describing the relationship between the polarization characteristics of reflected light at the interface of a medium and the angle of incidence, and the Brewster angle is a specific angle of incidence when the reflected light is in a completely linear polarized state. This inverse normal solution is only valid when the angle of incidence is less than the Brewster angle. Beyond this condition, the single-value mapping relationship between the polarization angle and the normal azimuth angle fails. First, the depth value corresponding to each pixel in the weak texture plane region of the depth image is extracted. Combined with the camera intrinsic parameter matrix of the mixed reality head-mounted device, the preliminary three-dimensional coordinates are calculated according to the inverse operation formula of the pinhole camera projection model. The inverse operation formula of the pinhole camera projection model is that the three-dimensional coordinate X is equal to the pixel x-coordinate minus the principal point x-coordinate. Multiplying by the depth value and dividing by the focal length, the 3D coordinate Y equals the pixel's ordinate minus the principal point's ordinate multiplied by the depth value and divided by the focal length; the 3D coordinate Z equals the depth value. This process is repeated to calculate the preliminary 3D coordinates of all pixels within the weakly textured plane region, forming a complete local 3D point cloud set. This point cloud set serves as the foundational data for fitting the initial plane. Subsequently, the least squares method is used to perform plane fitting on the local 3D point cloud set. Least squares plane fitting aims to solve for the optimal plane equation parameters, minimizing the sum of the squared distances from all 3D points to the plane. The general form of the plane equation is AX + BY + CZ + D equal to zero. After fitting the plane equation parameters ABC, the coordinates of the coarse normal vector are the vectors corresponding to parameters ABC. This vector is then normalized.Normalization involves dividing the vector coordinates by the vector's magnitude. The vector magnitude is equal to the square root of the sum of the squares of A, B, and C, ultimately yielding the initial plane's coarse normal vector. This coarse normal vector is used for subsequent calculations of the ray incident angle. The direction vector of the imaging ray is the positive Z-axis direction of the camera coordinate system, i.e., vector 001. The ray incident angle is the angle between the imaging ray direction vector and the coarse normal vector. The incident angle is calculated using the dot product formula: the cosine of the incident angle equals the dot product of the ray direction vector and the coarse normal vector divided by the product of their magnitudes. Since both the ray direction vector and the coarse normal vector are unit vectors with a magnitude of 1, the cosine of the incident angle is directly equal to their dot product. The angle of incidence is calculated using the inverse cosine function. Simultaneously, based on the material type of the weakly textured planar region, the pre-calibrated Brewster angle values ​​for the corresponding material are retrieved. The Brewster angle is determined by the material's refractive index. Brewster angle calibration data for common weakly textured materials such as matte plastic, solid-color walls, wood panels, and metallic paint are stored. After preparing the angle of incidence and Brewster angle values, the applicable condition determination and pixel selection stage begins. The calculated angle of incidence value for each pixel is compared with the corresponding Brewster angle value for the material. It is then determined whether the polarization angle measurement value of each pixel within the local polarization angle block meets the applicable condition of the Fresnel reflection model where the angle of incidence is less than the Brewster angle. If the angle of incidence value of a certain pixel... If the incident angle of a pixel is greater than or equal to the Brewster angle value, the measured polarization angle of that pixel is determined not to meet the model's applicability conditions, and the pixel is marked as invalid. Its polarization angle value will not participate in the subsequent lookup table matching and azimuth candidate value statistics process. If the incident angle of a pixel is less than the Brewster angle value, the measured polarization angle of that pixel is determined to meet the model's applicability conditions, and the pixel is marked as valid. Its polarization angle value is retained for subsequent lookup table and statistical operations. After completing the validity determination of all pixels, only the polarization angle values ​​of valid pixels are retained for local polarization angle blocks, completely eliminating abnormal polarization angle data caused by incident angles exceeding the model's applicability range. For example, in a weakly textured planar area with a matte plastic material, the Brewster angle value is less than the Brewster angle value. The angle is 56 degrees. A pixel's calculated incident angle is 62 degrees, which is greater than Brewster's angle. This pixel is marked as invalid, and its polarization angle value is discarded. Another pixel has an incident angle of 43 degrees, which is less than Brewster's angle. It is marked as a valid pixel, and its polarization angle value participates in subsequent calculations. The entire correction relies on 3D geometric projection and optical physical constraints, without subjective parameter adjustments. It follows the applicable conditions of the Fresnel reflection model, ensuring that all polarization angle data participating in the inverse normal solution have physical validity. This significantly improves the accuracy of the inverse solution of the surface normal azimuth component, providing reliable data support for subsequent unit normal transformation and one-dimensional virtual constraint edge construction. It further enhances the pose drift suppression capability of mixed reality spatial positioning in weakly textured planar scenes.

[0030] After the azimuth component of the surface normal vector is inversely solved and converted into a unit normal vector in the camera coordinate system, the construction of one-dimensional virtual constraint edges is immediately initiated. This process, through standardized steps of cross-frame normal matching, camera displacement projection, residual definition, and information matrix construction, transforms the normal geometric constraints of the weakly textured plane into one-dimensional virtual constraints in pose graph optimization. Only the translational deviation of the plane normal is constrained, while the tangential degree of freedom is preserved. This suppresses camera pose drift in weakly textured scenes, compensates for the failure of traditional visual constraints in weakly textured regions, and improves the continuity and stability of mixed reality spatial positioning. The unit normal vector of the current frame is the normalized surface normal vector of the weakly textured plane region in the camera coordinate system corresponding to the current image frame, with a magnitude of 1, representing... The spatial orientation of the weakly textured plane in the current frame; the pose graph is a graph optimization data structure used in mixed reality spatial localization to store camera keyframe poses, spatial constraints, and map point observation data; keyframes are representative image frames selected from consecutive image frames for pose graph optimization and map construction; the previous keyframe is the most recent historical keyframe that has completed pose optimization and data storage; camera pose is a six-degree-of-freedom parameter describing the camera's position and orientation in the world coordinate system, including 3D position coordinates and 3D rotation quaternions; camera center position coordinates are the 3D coordinates (XYZ) of the camera's optical center in the world coordinate system, representing the camera's translation state; the difference vector is the distance from the camera center position coordinates of the previous keyframe to the current keyframe. The three-dimensional vector representing the center position coordinates of the camera in each frame represents the overall translational change of the camera between two frames. The projection length is the component length of the difference vector in a specified direction, here it is the translational component along the unit normal direction of the current frame. The translational deviation is the unreasonable displacement of the camera along the weak texture plane normal between two frames, serving as the residual term in pose graph optimization. The residual is the difference between the actual observed value and the theoretical estimate of the constraint term in pose graph optimization, used to measure the pose estimation error. The information matrix is ​​the weight matrix representing the confidence of the constraint term in pose graph optimization, determining the degree of influence of the constraint term on the optimization solution. The one-dimensional virtual constraint edge is a virtual constraint edge that only constrains a single degree of freedom, acting only on the translational degree of freedom in the plane normal, without affecting the tangential translation and rotational degrees of freedom. The surface normal vector obtained by inverse kinematics and normalization of the current frame is formally denoted as the unit normal vector of the current frame. This vector is stored in the feature data structure of the current frame and is associated with and bound to the camera pose, image data, and depth data of the current frame. Subsequently, a historical keyframe retrieval and matching operation is performed from the pose graph history database of mixed reality spatial localization. The retrieval rule is to filter out historical keyframes that observe the same weakly textured plane region as the current frame, select the keyframe with the closest timestamp to the current frame as the previous keyframe, retrieve the pose data structure of the previous keyframe, and extract and store the unit normal vector obtained by inverse kinematics for the same weakly textured plane region under the camera pose of the previous keyframe. This vector corresponds to the same physical plane as the unit normal vector of the current frame.To ensure consistency of cross-frame normal constraints, after obtaining the normal vector, extract the camera center coordinates from the current frame's camera pose, in 3D numerical form (Xc, Yc, Zc). Simultaneously, extract the camera center coordinates from the previous keyframe's camera pose, in 3D numerical form (Xp, Yp, Zp). Calculate the difference vector from the previous keyframe to the current frame's camera center coordinates. The difference vector is calculated by subtracting the X component of the previous keyframe's camera center coordinates from the current frame's X component, obtaining the X component of the difference vector; subtracting the Y component of the previous keyframe's camera center coordinates from the current frame's Y component, obtaining the Y component of the difference vector; and subtracting the previous keyframe's camera center coordinates from the Z component of the current frame's Z component, obtaining the Y component of the difference vector. The Z-component of the camera center position coordinates in the keyframe is used to obtain the Z-component of the difference vector. The difference vector fully represents the 3D translational change of the camera between two frames. Then, the projection length of this difference vector along the unit normal vector direction of the current frame is calculated. The calculation of the projection length follows the vector dot product rule. Specifically, the difference vector is multiplied by the unit normal vector of the current frame. Since the unit normal vector of the current frame is a unit vector with a magnitude of 1, the result of the dot product is directly equal to the projection length of the difference vector along the unit normal vector direction. This projection length is defined as the translational deviation. The physical meaning of the translational deviation is the non-rigid displacement generated by the camera along the normal of the weak texture plane. The weak texture plane is a rigid physical plane. Theoretically, the relative displacement of the camera along the plane normal... The translation should remain constant. The translation deviation is the error between the actual pose estimation and the theoretical rigid constraint. After defining the translation deviation, the construction of the one-dimensional virtual constraint edge begins. First, an information matrix of the constraint edge is constructed based on the unit normal vector of the current frame. The information matrix is ​​a three-dimensional diagonal matrix. The matrix construction rule is to assign a preset Gaussian weight to the normal direction corresponding to the unit normal vector of the current frame. The Gaussian weight is a confidence weight determined through standard weak texture scene calibration experiments. The value reflects the reliability of the normal constraint. In the two tangential directions perpendicular to the normal, the diagonal elements of the information matrix are assigned zero. A zero value means that the constraint edge has no constraint effect on the tangential degree of freedom, and only retains the constraint effect of the single normal dimension. Finally, the translation deviation is used as the one-dimensional virtual constraint edge. The residual term of the constraint edge uses the information matrix constructed based on the unit normal vector of the current frame as the weight term of the constraint edge, completing the binding configuration of residual and weight, and generating a complete one-dimensional virtual constraint edge. This constraint edge, as a new constraint term in pose graph optimization, connects the pose nodes of the current frame and the previous keyframe, and only applies constraints to the translational deviation of the plane normal. For example, if the unit normal vector of the current frame is 001, the camera center coordinates of the previous keyframe are 102030, the camera center coordinates of the current frame are 102035, the difference vector is 005, the projection length is 5, and the translational deviation is 5, the information matrix is ​​assigned a Gaussian weight of 0.8 in the Z-axis direction and 0 in the XY-axis direction, finally constructing a one-dimensional virtual constraint edge that only constrains the translation of the Z-axis normal, retaining only one-dimensional effective constraints.This approach avoids pose distortion caused by excessive constraints while suppressing normal drift, providing efficient weak texture constraints to support subsequent joint optimization of the pose graph.

[0031] After the one-dimensional virtual constraint edges are constructed and the residuals and information matrix are defined, the mixed reality spatial positioning initiates a pose graph optimization solution based on multi-source constraint fusion. This approach organically integrates traditional visual reprojection constraints, depth point cloud registration constraints, and one-dimensional virtual constraints on weakly textured planes. Through multi-constraint joint nonlinear least squares optimization, it suppresses the camera pose drift problem on the normal of the weakly textured plane, overcoming the technical bottlenecks of traditional mixed reality spatial positioning in weakly textured and textureless scenes where positioning fails and pose continuously shifts. This comprehensively improves the continuity, stability, and scene adaptability of spatial positioning. The pose graph optimization framework is a spatial positioning solution architecture built on graph optimization theory. This framework organizes all positioning-related data in the form of a graph structure, where nodes represent... The table represents the state variables to be optimized, and the edges represent the constraint observation conditions. The optimal camera pose is solved by minimizing the weighted sum of squared residuals of all constraint edges, which is the mainstream optimization scheme in the current field of vision and fusion localization. The state variable node representing the current camera pose is the node in the pose graph that represents the six-degree-of-freedom pose of the camera in the current frame. The six-degree-of-freedom pose includes two types of parameters: three-dimensional translation coordinates and three-dimensional rotation quaternions. The three-dimensional translation coordinates represent the spatial position of the camera in the world coordinate system, and the three-dimensional rotation quaternions represent the spatial attitude of the camera. These are the variables to be solved in the pose graph optimization. First, three different types of constraint edges are simultaneously connected to this state variable node. These three types of constraint edges provide constraints from three dimensions: visual features, three-dimensional geometry, and weakly textured plane normal direction, respectively, forming... The full-dimensional pose optimization constraint system uses visual reprojection error as the first constraint edge. Visual reprojection error is the basis for constraints in traditional visual simultaneous localization and mapping (VSL). Map points are static 3D spatial feature points reconstructed from multi-frame image observations in mixed reality. They are fixed spatial landmarks in the scene and serve as references for visual positioning. The visual reprojection error constraint edge connects the current camera pose state variable node with other keyframe nodes that have observed the same map point. Keyframes are spatially representative image frames selected from continuously acquired image frames, used for pose graph construction and map updating. Common observations of the same map point by different keyframes can form cross-frame closed-loop constraints, effectively suppressing pose accumulation errors. The residual of this constraint edge is... The reprojection error of map points on the current frame image is calculated as follows: The 3D map points in the world coordinate system are transformed to the camera coordinate system using rotation quaternions and translation coordinates based on the current camera pose. Then, combined with the camera intrinsic parameter matrix, these coordinates are projected onto the image plane of the current frame to obtain the theoretical projected pixel coordinates of the map points. The difference between these theoretical projected pixel coordinates and the actual observed map point pixel coordinates in the current frame is calculated; the two-dimensional norm of this difference is the reprojection error. This residual directly characterizes the degree of matching between the projected map point result and the actual observation result under the current camera pose. This residual constrains the full degrees of freedom of rotation and translation of the camera pose, ensuring spatial consistency of visual positioning. The second type of constraint edge is the depth point cloud ICP registration error constraint edge.A depth point cloud is a set of three-dimensional points projected from the depth value of each pixel in a depth image using camera intrinsic parameters. It represents the surface geometry of a scene. ICP registration, or Iterative Nearest Point Registration, is a classic algorithm in the field of 3D point cloud registration, used to calculate the optimal pose transformation between two sets of point clouds. A local point cloud map is a real-time constructed 3D point cloud map of a local scene in mixed reality, integrating static scene geometry information within the current field of view. It serves as a reference benchmark for depth geometric localization. The connection objects of the depth point cloud ICP registration error constraint edge are the state variable nodes of the current camera pose and the local point cloud map. The residual of this constraint edge is the distance error when registering the current frame depth point cloud with the local point cloud map. The specific calculation process of the distance error is to convert the current frame depth point cloud into a local point cloud map. The depth point cloud is transformed to the world coordinate system through the current camera pose. For each point in the current frame's depth point cloud, the nearest neighbor matching point in the local point cloud map is found. The Euclidean distance between all matching point pairs is calculated, and the arithmetic mean of all distances is used as the registration distance error. This residual represents the degree of geometric matching between the depth point cloud and the local scene map under the current camera pose, mainly constraining the translational degrees of freedom of the camera pose and improving the accuracy of 3D geometric positioning. The third type of constraint edge is a one-dimensional virtual constraint edge constructed in the early stage. The object connected by this constraint edge is the state variable node of the current camera pose and the previous keyframe node. The previous keyframe is the historical keyframe that is closest to the current frame in time and has completed pose optimization and data storage. The residual of this constraint edge is along the weak texture plane. The translational deviation in the direction of the unit normal vector of the plane is the non-rigid displacement error of the camera along the plane normal between two frames. The constraint edges only impose constraints on the normal translation degree of freedom, while the tangential translation and all rotation degrees of freedom are unconstrained. It is a directional constraint specifically designed for weakly textured planar scenes. After completing the connection of all three types of constraint edges, the joint optimization solution stage is entered. The overall goal of the optimization solution process is to minimize the weighted sum of squares of the residuals of all constraint edges. The specific calculation method of the weighted sum of squares is to multiply the residual vector of each constraint edge by the information matrix of the corresponding constraint edge, and then multiply by the transpose of the residual vector to obtain the weighted residual square term of a single constraint edge. The weighted residual square terms of all constraint edges are accumulated to obtain the overall objective function value, where one The weights of the virtual constraint edges are determined by their dedicated information matrix. This information matrix assigns preset Gaussian weights to the plane normal direction. The values ​​of these Gaussian weights are dynamically determined by the area of ​​the weakly textured plane region, boundary sharpness, and statistical consistency of the polarization angle. The weights in the tangential direction are set to zero to ensure the one-dimensional characteristic of constraining only the normal translation. The Levenberg-Marquardt algorithm is used for iterative optimization. The Levenberg-Marquardt algorithm is a nonlinear least squares optimization algorithm that combines the advantages of gradient descent and Gauss-Newton methods. It possesses good local optimum avoidance ability and convergence stability, making it suitable for the nonlinear solution requirements of camera pose optimization. During the algorithm iteration, the values ​​of the current camera pose state variables, namely the 3D translation coordinates and the 3D rotation quaternions, are continuously adjusted.Each iteration calculates the Jacobian and Hessian matrices of the residuals of all current constraint edges, updates the camera pose state variables, and then recalculates the overall objective function value to determine if the optimization process has converged. The convergence condition is set as follows: the change in the overall objective function value over five consecutive iterations is less than a preset convergence threshold, or the number of iterations reaches the set maximum number of iterations. When the optimization process meets the convergence condition, the iteration terminates. The output of the converged current camera pose at this point has undergone drift suppression on the unit normal direction of the weak texture plane by one-dimensional virtual constraint edges, correcting the pose deviation of traditional positioning in weak texture regions. This is the final corrected camera pose after unit normal direction drift suppression. This corrected camera pose integrates full-dimensional constraints of visual features, depth geometry, and weak texture normals, maintaining stable spatial positioning in weak texture scenes where traditional positioning fails, such as solid-color walls, smooth desktops, and matte floors, thus comprehensively improving the spatial positioning performance of mixed reality headsets.

[0032] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A mixed reality spatial localization method based on multi-source image fusion, characterized in that: Includes the following steps: S1. Synchronously acquire RGB images, depth images and polarization original images of the scene through a mixed reality head-mounted device, calculate and generate a polarization angle map from the polarization original image, and perform time stamp alignment and geometric calibration on the RGB images and depth images to establish a pixel-by-pixel mapping relationship. S2. The corner detection algorithm is used to extract feature points from the RGB image. Continuous image regions with feature point response values ​​lower than a preset threshold and point cloud curvature less than the curvature threshold after depth image verification are marked as weak texture plane regions. The boundaries of the weak texture plane regions are segmented based on the depth gradient information of the depth image. S3. For the weak texture plane region, extract the corresponding local polarization angle block from the polarization angle map. Based on the Fresnel reflection model, use the single-value mapping relationship between the polarization angle of the reflected light and the azimuth angle of the surface normal when the incident angle is less than the Brewster angle. Solve the azimuth angle component of the surface normal vector of the weak texture plane region in the camera coordinate system from the local polarization angle block. S4. Convert the azimuth component of the surface normal vector into a unit normal vector in the camera coordinate system, and construct a one-dimensional virtual constraint edge acting on the current camera pose. The one-dimensional virtual constraint edge is configured to use only the translation deviation between the current frame and the previous keyframe along the direction of the unit normal vector as the residual term. Its information matrix is ​​assigned a preset Gaussian weight in the normal direction and a value of zero in the tangential direction. S5. Add one-dimensional virtual constraint edges as new constraint terms, and combine them with visual reprojection error constraint edges and depth point cloud ICP registration error constraint edges into the pose graph optimization framework. Use the Levenberg-Marquardt algorithm for joint nonlinear least squares optimization to solve the problem and output the corrected camera pose after unit normal vector direction drift suppression.

2. The mixed reality spatial localization method based on multi-source image fusion according to claim 1, characterized in that: The process of generating a polarization angle map from the original polarization image specifically includes: By using a polarization camera integrated into a mixed reality head-mounted device, original polarized images of the same scene in multiple different polarization directions are acquired at the same time. Stokes parameters corresponding to each pixel are calculated based on the grayscale values ​​of the original polarized images. The polarization angle of the reflected light at each pixel is calculated using the first two parameters of the Stokes parameters. A polarization angle map aligned with the RGB image and depth image space is generated based on the polarization angle.

3. The mixed reality spatial localization method based on multi-source image fusion according to claim 2, characterized in that: After generating the polarization angle map, the method further includes spatial consistency filtering of the polarization angle map, specifically including: The polarization angle statistical features of each pixel in the polarization angle map within its local neighborhood are calculated. Based on these polarization angle statistical features, it is determined whether the polarization angle of the current pixel differs from that of the pixels in its neighborhood. If there is a difference exceeding a preset standard, the polarization angle of the current pixel is replaced with the mode represented by the polarization angle statistical features within its neighborhood. During the spatial consistency filtering process, the neighborhood size and difference judgment threshold of the spatial consistency filtering are adaptively adjusted based on the polarization angle consistency measure in the polarization angle statistical features.

4. The mixed reality spatial localization method based on multi-source image fusion according to claim 1, characterized in that: The step of marking continuous image regions with feature point response values ​​lower than a preset threshold and point cloud curvature less than a curvature threshold after depth image verification as weakly textured plane regions specifically includes: Corner detection is performed on the RGB image to obtain feature points, and the response value of each feature point is calculated. All feature point positions with response values ​​lower than the preset threshold are mapped to the corresponding depth image coordinates, and the depth value is obtained from the depth image through the depth image coordinates. Then, based on the depth value and the surrounding depth value, the curvature of the 3D point cloud corresponding to the feature point position in the local range is calculated. Pixel positions with feature point response values ​​lower than the preset threshold but whose corresponding point cloud curvature is not less than the curvature threshold are removed. Finally, the set of spatially connected pixels in the depth image that are not removed and are surrounded by the removed pixel positions are marked as weak texture plane regions.

5. A mixed reality spatial localization method based on multi-source image fusion according to claim 4, characterized in that: The step of segmenting the boundaries of weakly textured planar regions based on the depth gradient information of the depth image specifically includes: Calculate the depth gradient magnitude within each weak texture plane region. Within the weak texture plane region, starting from any pixel, perform ray scanning along multiple directions. When the depth gradient magnitude on the scanning path first exceeds a gradient threshold, record the endpoint of the scanning path as a candidate boundary point. Collect candidate boundary points in all directions and connect them using a curve fitting method to form the preliminary closed boundary of the weak texture plane region. Then, smooth the preliminary closed boundary and adjust the candidate boundary points using depth values ​​to obtain the final boundary of the weak texture plane region.

6. The mixed reality spatial localization method based on multi-source image fusion according to claim 1, characterized in that: The Fresnel reflection model utilizes the single-valued mapping relationship between the polarization angle of reflected light and the azimuth angle of the surface normal when the incident angle is less than the Brewster angle. The azimuth component of the surface normal vector of the weakly textured planar region in the camera coordinate system is obtained by inversely solving local polarization angle blocks. Specifically, this includes: A mapping lookup table is pre-established through calibration. During inverse kinematics, for each pixel's polarization angle value within the local polarization angle block extracted from the polarization angle map, the corresponding surface normal vector azimuth angle candidate value is searched in the mapping lookup table. The azimuth angle candidate values ​​obtained for all pixels within the local polarization angle block are statistically analyzed, and the azimuth angle candidate value with the highest frequency of occurrence is selected as the azimuth angle component of the surface normal vector of the weak texture plane region in the camera coordinate system.

7. A mixed reality spatial localization method based on multi-source image fusion according to claim 6, characterized in that: Before performing the inverse solution, the method further includes correcting the local polarization angle block, specifically including: Using the depth information provided by the depth image, and combined with camera intrinsic parameters, the preliminary three-dimensional coordinates of the three-dimensional point corresponding to each pixel in the weak texture plane region in the camera coordinate system are calculated. An initial plane is fitted based on the preliminary three-dimensional coordinates, and a coarse normal vector of the initial plane is calculated. The incident angle of light is calculated based on the coarse normal vector to determine whether the polarization angle measurement value of each pixel in the local polarization angle block meets the applicable condition of the Fresnel reflection model that the incident angle is less than the Brewster angle. For pixels that do not meet the applicable condition, their polarization angle values ​​are not included in the mapping relationship lookup table mapping and statistical process.

8. A mixed reality spatial localization method based on multi-source image fusion according to claim 1, characterized in that: The construction of a one-dimensional virtual constraint edge acting on the current camera pose specifically includes: The unit normal vector is denoted as the unit normal vector of the current frame. From the historical data of the pose graph, the unit normal vector for the same weak texture plane region under the camera pose of the previous keyframe is obtained. The difference vector of the camera center position coordinates from the previous keyframe to the current frame is calculated. The projection length of the difference vector in the direction of the unit normal vector of the current frame is calculated. The projection length is defined as the translation deviation. Finally, the translation deviation is used as the residual, and the one-dimensional virtual constraint edge is constructed using the information matrix constructed by the unit normal vector of the current frame.

9. A mixed reality spatial localization method based on multi-source image fusion according to claim 8, characterized in that: The information matrix is ​​assigned a preset Gaussian weight in the normal direction, specifically including: The specific value of the Gaussian weight is dynamically adjusted based on the attributes of the current weak texture plane region, including the area size, boundary clarity, and statistical consistency of the current weak texture plane region.

10. A mixed reality spatial localization method based on multi-source image fusion according to claim 8, characterized in that: The step involves adding one-dimensional virtual constraint edges as new constraint terms, and incorporating them together with visual reprojection error constraint edges and depth point cloud ICP registration error constraint edges into the pose graph optimization framework. A joint nonlinear least squares optimization solution is then performed using the Levenberg-Marquardt algorithm, specifically including: In the pose graph optimization framework, three types of constraint edges are connected to the state variable node of the current camera pose. The first type is the visual reprojection error constraint edge, which connects the state variable node of the current camera pose with other keyframe nodes that observe the same map point. The residual is the reprojection error of the map point on the current frame image. The second type is the depth point cloud ICP registration error constraint edge, which connects the state variable node of the current camera pose with the local point cloud map. The residual is the distance error when registering the current frame depth point cloud with the local point cloud map. The third type is the one-dimensional virtual constraint edge, which connects the state variable node of the current camera pose with the previous keyframe node, and the residual is the translation deviation along the unit normal vector direction. The optimization solution process aims to minimize the weighted sum of squares of the residuals of all constraint edges. The weights of the one-dimensional virtual constraint edges are determined by the information matrix. The Levenberg-Marquardt algorithm is used to iteratively adjust the values ​​of the camera pose state variables until the overall objective converges. The final output of the converged current camera pose is the corrected camera pose after unit normal vector direction drift suppression.

Images