Spatial Computation Point Cloud Data Acquisition and Practical Training System Based on AR Glasses

Abstract

The application is specifically a space calculation point cloud data collection and practical teaching system based on AR glasses, relates to the technical field of three-dimensional point cloud processing, and comprises: an eye movement tracking and line of sight vector solving module; a point cloud feature extraction and curvature evaluation module; a view-dependent dynamic grid contraction module; a non-uniform topological geometry interpolation reconstruction module; and an AR low-delay adaptive hybrid rendering module.In the application, a full-process processing link is constructed based on pure three-dimensional differential geometry and linear algebra algorithms, the physiological characteristics of the foveal vision of the human eye are directly mapped to the bottom layer point cloud topological processing, the view-dependent non-uniform grid contraction is realized through a multi-dimensional weight fusion mechanism, and the algorithmic load of the redundant area of the visual field is reduced while the key geometric details of the practical training are completely retained in the core area of the user's vision.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud processing technology, and in particular to a spatial computing point cloud data acquisition and training system based on AR glasses. Background Technology

[0002] Industrial practical training is a core component of talent cultivation in fields such as high-end equipment manufacturing, precision machining, and electrical maintenance. Traditional training models suffer from drawbacks such as high equipment costs, significant operational risks, limited teaching scenarios, and low standardization. Augmented reality (AR) technology can precisely integrate virtual training guidance information with real physical space, providing a low-risk, highly immersive, and repeatable solution for practical training, thus becoming an important development direction in the field of industrial practical training.

[0003] The core technological bottleneck of current AR training systems lies in the real-time processing and rendering of 3D spatial point cloud data. As mobile devices, AR glasses have strict physical limits on GPU computing power and memory bandwidth. However, depth sensors in industrial training scenarios can generate millions of high-frequency point cloud data points. Traditional processing techniques cannot achieve low-latency, high-fidelity point cloud rendering on mobile devices, and the following areas require improvement:

[0004] Existing technologies generally use global uniform voxel downsampling technology to process point cloud data. In order to meet real-time requirements, key geometric details of the training scenario will be lost indiscriminately, resulting in distortion of the core training structure such as gears, welds, and sharp edges of precision equipment.

[0005] The gaze vector calculated by existing eye-tracking technology is easily affected by micro-saccades and sensor noise, resulting in high-frequency jitter and insufficient accuracy. It cannot provide a stable and accurate geometric origin for non-uniform space calculations, thus limiting the practical application of viewpoint-dependent rendering technology.

[0006] Therefore, a spatial computing point cloud data acquisition and training system based on AR glasses is proposed to address the aforementioned problems. Summary of the Invention

[0007] The purpose of this invention is to provide a spatial computing point cloud data acquisition and training system based on AR glasses to solve the above-mentioned problems.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A spatial computing point cloud data acquisition and training system based on AR glasses includes:

[0010] The eye-tracking and gaze vector calculation module is configured to acquire the user's grayscale image of the eyes, extract the pupil edge and fit the geometric center, combine the six-degree-of-freedom spatial pose data of the AR glasses and the Kalman filter, calculate and output the absolute gaze direction vector and the absolute gaze center point in the three-dimensional world coordinate system.

[0011] The point cloud feature extraction and curvature evaluation module is configured to construct a multi-dimensional spatial binary tree from the original discrete point cloud acquired by the depth sensor to perform local nearest neighbor retrieval, construct the covariance matrix through principal component analysis and solve the eigenvalues, and extract the local surface change rate scalar as the geometric curvature weight.

[0012] The viewpoint-dependent dynamic mesh shrinking module is configured to receive the absolute gaze direction vector, the absolute gaze center point, and the geometric curvature weights. It calculates the eccentricity angle of each point relative to the gaze center axis and maps it to Gaussian visual importance weights. Then, it combines the depth distance weights to generate a comprehensive retention importance. It uses the fourth-order symmetric basic error quadratic matrix based on importance to evaluate the edge folding cost and perform dynamic downsampling shrinking.

[0013] The non-uniform topological geometry interpolation reconstruction module is configured to receive the shrunken non-uniform point cloud, construct a moving least squares surface fitting based on the inverse proportional adaptive smoothing factor that comprehensively retains importance, use the adaptive rolling sphere method with the radius dynamically constrained by importance to weave a triangular mesh, and repair closed holes through dynamic programming to generate a non-uniform watertight manifold mesh model.

[0014] The AR low-latency adaptive hybrid rendering module is configured to receive a non-uniform watertight manifold mesh model and push it into the video memory. It performs variable-rate fragment shading in combination with the absolute gaze center point, and records the real mesh depth value by writing a mask through only opening the depth buffer. This allows for depth testing and physical-level occlusion culling of the virtual training model. Before output, it uses an asynchronous time-warped thread to perform reprojection forward prediction integration.

[0015] Preferably, after acquiring the grayscale image of the eye, the eye-tracking and gaze vector calculation module extracts the set of pixels at the edge of the pupil through edge detection and constructs a conic section algebraic equation. To fit the coordinates of the pupil center, where, and These are the pixel coordinates on the two-dimensional image plane. , , , , , All of these are algebraic coefficients of the algebraic equation of the conic section, and the algebraic coefficients satisfy the discriminant. The constraints are set to ensure that the fitting result is a closed ellipse.

[0016] Preferably, the method further includes:

[0017] The Kalman filter uses a six-dimensional column vector as the state vector, which contains three position coordinate components of the absolute gaze center point in three-dimensional space, and three movement rate components on the corresponding orthogonal spatial axes.

[0018] Preferably, when solving for eigenvalues, the point cloud feature extraction and curvature evaluation module performs singular value decomposition on the covariance matrix to obtain three non-negative eigenvalues ​​arranged in descending order, and uses the formula... Calculate the local surface change rate scalar;

[0019] in, The scalar representing the rate of change of the local surface. , , These represent the three non-negative eigenvalues ​​of the covariance matrix after decomposition, arranged in descending order.

[0020] Preferably, the eigenvector corresponding to the extracted minimum eigenvalue is used as the surface normal vector of the locally fitted surface, and the inner product constraint formula is forcibly satisfied. ,in, Represents the surface normal vector. The three-dimensional absolute coordinates representing the absolute viewpoint emission source. It represents the three-dimensional coordinates of local spatial points in the original discrete point cloud, so as to achieve global consistency of the normal field of the entire point cloud.

[0021] Preferably, the viewpoint-dependent dynamic mesh shrinkage module solves for the eccentricity angle by using the inverse cosine function between the observation ray vector from the observer to the discrete point and the absolute line-of-sight direction vector, and then substitutes the eccentricity angle into a Gaussian visual importance weighting function. In, among them, Represents discrete points Gaussian visual importance weights Represents discrete points Compared to the eccentricity angle of the line of sight's central axis, This represents the visual attenuation control parameter.

[0022] Preferably, in the viewpoint-dependent dynamic mesh shrinkage module:

[0023] When performing edge folding cost evaluation, the edge folding cost function is: ;

[0024] in, Represents discrete points and discrete points The folding cost of the connecting edges formed, This represents the new point generated by the folding and is a four-dimensional homogeneous coordinate column vector. The row vector representing the transpose of the new point. and Representing discrete points and discrete points The overall importance of retention and They respectively represent the points assigned to discrete points and discrete points The fourth-order symmetric fundamental error quadratic form matrix, Represents discrete points and discrete points The Euclidean distance between them.

[0025] Preferably, in the non-uniform topological geometry interpolation reconstruction module:

[0026] The inverse proportional adaptive smoothing factor of the moving least squares surface fitting satisfies the formula ,in Representative point The inverse proportional adaptive smoothing factor at the location, Represents the reference adjustment constant;

[0027] The adaptive rolling ball method satisfies the equation ,in Represents spatial coordinates The radius of the virtual sphere at that location, Represents the minimum contact radius of the base. Represents the maximum expansion rate constant. Represents spatial coordinates The overall importance of retention at each location This represents a factor that is sensitive to the decay of importance.

[0028] Preferably, the AR low-latency adaptive hybrid rendering module performs the following during depth testing:

[0029] The color write mask for the non-uniform watertight manifold mesh model is turned off while the depth write mask remains on, so that the real mesh depth value can be extracted and stored in the global depth buffer. When the depth value of the virtual training model fragment is greater than the real mesh depth value at the same screen coordinate in the global depth buffer, the hardware performs physical-level occlusion culling.

[0030] Preferably, the method further includes:

[0031] The asynchronous time-warping thread runs independently of the main rendering thread. Before the rendered image is output, it captures the transient angular velocity vector of the inertial measurement unit and uses Taylor series expansion to predict the latest rotational attitude quaternion.

[0032] The prediction equation is: ;

[0033] in The latest rotational attitude quaternion is represented. Represents the transient angular velocity vector. The magnitude of the transient angular velocity vector. It represents the scalar value representing the time difference from the start of rendering to the output of the image. This represents the head rotation quaternion during sampling by the main rendering thread. This represents the quaternion multiplication operator and performs reverse geometric reprojection on the output image based on the predicted attitude deviation.

[0034] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0035] 1. This invention constructs a complete processing link based on pure three-dimensional differential geometry and linear algebra algorithms, directly mapping the physiological characteristics of human foveal vision to the underlying point cloud topology processing. Through a multi-dimensional weight fusion mechanism, it achieves viewpoint-dependent non-uniform mesh shrinkage, which fully preserves the key geometric details of the training in the core area of ​​the user's vision while reducing the computational load of redundant areas of the field of vision.

[0036] 2. This invention achieves physical-level virtual-real space occlusion fusion through end-to-end deep optimization of the underlying rendering pipeline and phantom mesh depth masking technology, solving the clipping problem between virtual training models and the real environment and operator limbs; combined with hardware-level variable rate shading and asynchronous time warp forward prediction technology, it significantly reduces the rendering load without reducing the user's subjective visual clarity, compresses the end-to-end delay from motion to imaging, and avoids dizziness caused by visual and vestibular conflict. Attached Figure Description

[0037] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0038] Figure 1 This is a system structure diagram of the present invention. Detailed Implementation

[0039] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0040] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0041] Example 1

[0042] Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0043] Appendix Figure 1 The diagram shows the structural block diagram of the spatial computing point cloud data acquisition and training teaching system based on AR glasses provided in the embodiments of the present invention. It illustrates the connection relationship between the eye tracking and gaze vector calculation module and the AR low-latency adaptive hybrid rendering module, and marks the main functional interaction flow of each module.

[0044] In this embodiment, it includes:

[0045] The eye-tracking and gaze vector calculation module is configured to acquire the user's grayscale image of the eyes, extract the pupil edge and fit the geometric center, combine the six-degree-of-freedom spatial pose data of the AR glasses and the Kalman filter, calculate and output the absolute gaze direction vector and the absolute gaze center point in the three-dimensional world coordinate system.

[0046] This module is configured to capture the user's visual physiological characteristics through a high-frequency hardware sensor array, and output a line-of-sight ray vector that is absolutely smooth and has no delay in a three-dimensional world coordinate system based on spatial geometric mapping and temporal filtering algorithms.

[0047] This vector is the geometric origin for all subsequent calculations in non-uniform space.

[0048] On the inner frame of the AR glasses' lens, this system has multiple near-infrared light-emitting diodes (NIRLEDs, with emission wavelengths typically between 850 and 940 nanometers to avoid interference from visible light on the user's vision) arranged in a ring array. When the infrared beam shines on the operator's eyeball, it will cause specular reflection on the outer surface of the cornea to form the first Pullfield image (i.e., corneal reflection spot). At the same time, the infrared light penetrates the cornea into the anterior chamber, is absorbed by the pupil to appear as an extremely dark area, and is diffusely reflected by the iris to appear as a gray-white area.

[0049] The AR glasses' built-in global exposure infrared miniature camera captures sequences of two-dimensional grayscale images of the eye at an extremely high overclocked frame rate of 120 Hz to 240 Hz. To eliminate interference from sensor thermal noise and ambient stray infrared light, the system first performs a convolutional smoothing operation based on a two-dimensional Gaussian kernel on the original two-dimensional grayscale image in the microprocessor's digital signal processing unit. Subsequently, the system performs adaptive histogram equalization to enhance the grayscale gradient contrast between the dark area of ​​the pupil and the bright area of ​​the iris.

[0050] After image preprocessing, the system uses an improved nonmaximum suppression Canny edge detection operator to extract the subpixel-level physical contour of the pupil. Because eye movement causes perspective distortion of the originally circular pupil on the 2D camera target surface, its projection is strictly represented as a 2D ellipse.

[0051] Assume the effective pixel set of the pupil edge extracted by the system is The system constructs the general form of the algebraic equations for conic sections:

[0052] ;

[0053] To ensure that the above algebraic equation fits a closed ellipse in a physical sense, rather than a divergent hyperbola or parabola, the system introduces extremely strict mathematical constraints, namely, the discriminant of the conic section must satisfy: .

[0054] in, and These are the pixel coordinates on the two-dimensional image plane. , , , , , All of these are the algebraic coefficients of the algebraic equation of the conic section;

[0055] The system utilizes the Lagrange multiplier method and generalized eigenvalue decomposition technique to solve the aforementioned constrained nonlinear least squares objective function, obtaining the coefficients. to The optimal analytical solution was obtained, and then the exact absolute coordinates of the geometric center of the pupil in the two-dimensional image plane were derived. Simultaneously, the system extracts the sub-pixel center coordinates of the corneal reflective spot by calculating the local gray-level extreme values ​​of the image. .

[0056] The system calculates the two-dimensional coordinate deviation vector. Pupil-corneal reflectance features are extracted. To map these two-dimensional features into a three-dimensional direction vector, the system pre-imports a calibration parameter matrix based on polynomial surface regression, nonlinearly mapping the two-dimensional bias into a three-dimensional gaze direction vector in the local eye-tracking coordinate system of the AR glasses. .

[0057] Because the user's head is in continuous motion in the training environment, the system must unify the local eye-tracking vectors into the global world coordinate system. The system obtains the six-degree-of-freedom (6DoF) spatial pose of the head at the current timestamp by frequently reading data from the microelectromechanical system (MEMS) six-axis inertial measurement unit (IMU) and the binocular visual simultaneous localization and mapping (V-SLAM) system built into the AR glasses. Let the head translation vector be... The three-dimensional rotation matrix is The rigid physical translation offset matrix of the eye-tracking camera relative to the SLAM coordinate origin is: Then the absolute viewpoint emission source coordinates in the world coordinate system. With absolute line-of-sight vector The solution constraints are:

[0058] ;

[0059] ;

[0060] To eliminate high-frequency visual jitter caused by physiological micro-saccades of the eyeball, a state machine-level Kalman filter is connected before the output beam.

[0061] The system defines a six-dimensional column vector as the system state vector of the Hidden Markov Model. The first three components represent the three-dimensional position of the gaze point in world space, and the last three components represent the movement rate of the gaze point along the three orthogonal spatial axes.

[0062] The system constructs a linear state transition matrix and observation matrix based on Newton's laws of kinematics, and dynamically iteratively calculates the Kalman optimal gain coefficient by real-time calculation of the observation noise covariance matrix and process noise covariance matrix. After optimal gain-weighted posterior state estimation, the system outputs a stationary three-dimensional spatial line-of-sight parameter ray equation:

[0063] ;

[0064] (in (This is a spatial depth scalar). The first geometric intersection of this ray with the 3D bounding box of the training physical environment is defined by the system as the user's current absolute gaze center point.

[0065] The point cloud feature extraction and curvature evaluation module is configured to construct a multi-dimensional spatial binary tree from the original discrete point cloud acquired by the depth sensor to perform local nearest neighbor retrieval, construct the covariance matrix through principal component analysis and solve the eigenvalues, and extract the local surface change rate scalar as the geometric curvature weight.

[0066] This module is configured to perform a quantitative evaluation of the microscopic topological morphology of discrete spatial point clouds collected by AR sensors using only three-dimensional differential geometry and linear algebra matrix theory, without relying on any high-performance deep learning black-box models, and to accurately separate the "high-frequency complex structures" and "low-frequency flat redundancies" in the training scenario.

[0067] The structured light or time-of-flight (ToF) depth sensor at the front of the AR glasses first generates an initial cloud of points containing millions of disordered discrete 3D coordinates. , where each discrete point The system first divides the world coordinate system into a uniform three-dimensional cubic voxel mesh, calculates the three-dimensional geometric centroid of all points falling inside each voxel, and replaces all discrete points within the voxel with this unique centroid, completing the first stage of rapid noise reduction and sparsification.

[0068] To perform microscopic surface analysis on each point after sparsification, the system constructs a multidimensional spatial binary tree data structure. By calculating the variance of the spatial dimensions and finding the median as the hyperplane split point on the dimension with the maximum variance, the system... The entire Kd-tree can be constructed in a time complexity of [time value missing]. Using this data structure, for any point in space... The system can extremely quickly retrieve its nearest neighbor using the Euclidean distance metric. Each neighboring point constitutes a local neighborhood point set. .

[0069] The system assumes a local neighborhood point set. The unknown points are implicitly distributed on a continuous two-dimensional differential manifold surface. The system uses principal component analysis (PCA) to inversely approximate the local geometric properties of this unknown surface. The system first calculates the spatial centroid of the local point set. .

[0070] Based on this centroid, the system constructs a third-order symmetric positive semi-definite covariance matrix for this local neighborhood. :

[0071] ;

[0072] This matrix profoundly reveals the variability of the local point cloud distribution in three-dimensional space. The system analyzes the matrix... Perform singular value decomposition (SVD) to solve the characteristic equation. (in (For a third-order identity matrix), three non-negative real eigenvalues ​​are obtained. (Strictly arranged in descending order) ) and its three corresponding orthogonal eigenvectors .

[0073] In discrete approximation in differential geometry, the minimum eigenvalue Corresponding feature vector This represents the direction in which the spatial distribution of the point cloud changes least in that local region. Mathematically, this direction is equivalent to the surface normal vector of the locally fitted surface. .

[0074] Because the sign of the eigenvector is ambiguous, the directly calculated normal vector may exhibit incorrect internal and external orientations. The system utilizes the absolute viewpoint emission source coordinates calculated by the first module. Introduce viewpoint topological constraints. The system mandates that the normal vector of each point must form an acute angle with the line-of-sight vector pointing from the point to the observer, i.e., the vector dot product constraint must be satisfied: If the inner product is detected to be less than zero, the system immediately performs an inversion operation on the normal vector. This ensures that the normal field of the entire point cloud has a globally consistent topological orientation facing the operator.

[0075] Subsequently, the system constructs a surface change rate scalar characterizing the degree of local undulation using three eigenvalues. :

[0076] ;

[0077] This parameter constitutes the core criterion for evaluating geometric features in this module. When the point cloud is on an extremely flat training platform, the dispersion in the normal direction is almost zero. Therefore However, when the point cloud is located in extremely complex training equipment such as gear meshing parts, circuit board solder joints, or sharp tool edges, the dispersion in the three directions is similar. This will significantly increase. The system extracts this surface change rate parameter as a quantitative index of geometric high-frequency features, assigning an absolutely objective geometric preservation weight to the subsequent non-uniform shrinkage algorithm. .

[0078] The viewpoint-dependent dynamic mesh shrinking module is configured to receive the absolute gaze direction vector, the absolute gaze center point, and the geometric curvature weights. It calculates the eccentricity angle of each point relative to the gaze center axis and maps it to Gaussian visual importance weights. Then, it combines the depth distance weights to generate a comprehensive retention importance. It uses the fourth-order symmetric basic error quadratic matrix based on importance to evaluate the edge folding cost and perform dynamic downsampling shrinking.

[0079] This module is configured to receive the 3D gaze vector and absolute gaze center point from the real-time high-frequency output of the first module, while simultaneously receiving the initial point cloud data with a surface change rate scalar output from the second module. The core breakthrough of this module lies in its departure from the currently widely used global uniform voxel downsampling technique. Instead, it directly maps the nonlinear attenuation characteristics of the fovea of ​​the human visual system onto the underlying 3D point cloud topology, constructing a dynamic downsampling algorithm that adaptively "dissolves" and "shrinks" from the gaze focal point towards the visual field edge.

[0080] The cone cells in the human retina are concentrated in the fovea of ​​the macula, which means that the human eye has the highest spatial resolution only within a very small field of view directly in front of it (usually a solid angle of two to five degrees), while visual acuity decreases non-linearly and exponentially towards the periphery. Based on this physiological principle, the system first considers any existing three-dimensional point in space. Perform visual eccentricity quantification.

[0081] The absolute coordinates of the viewpoint emission source output by the first module are known to be... The absolute line-of-sight direction vector is The system is constructed by pointing from the observer (i.e., the emission source) to this three-dimensional discrete point. Observation ray vector :

[0082] ;

[0083] Subsequently, the system rigorously solves for the eccentricity angle of the point relative to the current line-of-sight center axis using the inverse cosine function of the vector dot product in spatial geometry. :

[0084] ;

[0085] After calculating the eccentricity angle, the system constructs a nonlinear Gaussian visual importance weighting function. The function maps angle values ​​to a normalized probability distribution between zero and one, quantifying the absolute perceived importance of that point within the operator's current field of vision.

[0086] ;

[0087] in, It is strictly defined as a visual attenuation control parameter. The system dynamically adjusts this parameter by reading the precision requirements of the training subject currently being performed by the AR glasses; when the training subject is precision welding, Setting a very small value (such as a three-degree radian value) causes the weights of surrounding point clouds to decay rapidly; when the subject is large-scale mechanical assembly... Enlarge appropriately.

[0088] If downsampling is solely based on the visual eccentricity angle, the sharp edges of tools that are at the edge of the user's field of view but have extremely high physical significance for collision avoidance will be over-smoothed, resulting in severe clipping issues. Therefore, this system constructs a multi-dimensional weighted fusion equation.

[0089] The system first introduces the local surface change rate obtained from the second module. This is directly equivalent to or linearly mapped to geometric curvature weights. Subsequently, the system introduces depth-distance weights based on the principle of perspective projection. Because objects closer to the human eye form larger images on the retina, more topological details need to be preserved:

[0090] ;

[0091] in To prevent the program from crashing due to a denominator of zero, a very small positive constant (such as...) is used. ).

[0092] The system ultimately normalizes and linearly fuses the visual weights, geometric weights, and depth weights to generate points. Overall Retention Importance :

[0093] ;

[0094] in The normalized collaborative control coefficients, and strictly satisfy the following conditions: Through this equation, even if a point is located at the edge of the field of vision (low visual weight), its overall retention will still be dynamically increased as long as it is a sharp gear of the training equipment (extremely high geometric weight) or extremely close to the operator's face (extremely high depth weight).

[0095] After obtaining the overall importance of all points globally, the system executes the underlying mesh shrinkage algorithm. This system adopts the classic quadratic error metric algorithm in computer graphics, and has made significant modifications to the overall weights.

[0096] In the implicit triangle topology of a point cloud, for any line originating from point... and points Physical connection edges formed The system attempts to "fold" and merge it into a new center point. To perform rigorous matrix dimension operations, this system strictly requires that the generation of new points here be strictly controlled. Homogeneous coordinates must be used, that is, it must be defined as a four-dimensional column vector. .

[0097] Simultaneously, the system pre-calculates and assigns a four-by-four symmetric fundamental error quadratic form matrix to each initial vertex in the point cloud. .point Corresponding matrix ,point Corresponding matrix The system defines an improved weighted edge folding cost function. :

[0098] ;

[0099] The physical meaning of the above-mentioned extremely ingenious cost function is as follows: the quadratic term in the square brackets is used to ensure that the geometric shape deviation of the new point after folding is minimized; the distance norm term at the end is used to prioritize folding extremely close redundant points in the space; and the comprehensive weight term in the denominator causes points with high visual attention or high frequency geometric features (i.e., the denominator is extremely large) to generate extremely high folding costs.

[0100] The system calculates the folding cost of all edges in the computation space and pushes it into a min-heap of a binary tree structure. A background thread continuously pops the edge with the lowest cost from the top of the min-heap, performs physical folding, updates the coordinates and covariance matrix of the new point, and dynamically refreshes the folding cost of the remaining edges connected to that new point. This iterative process runs at a microsecond-level speed until the total number of vertices in the entire point cloud scene decreases to the maximum throughput threshold allowed by the current AR glasses' GPU memory bandwidth. Thus, the point cloud maintains a density of millions in the gaze area, instantly shrinking to a coarse skeleton of tens of thousands at the edge of the field of view.

[0101] The non-uniform topological geometry interpolation reconstruction module is configured to receive the shrunken non-uniform point cloud, construct a moving least squares surface fitting based on the inverse proportional adaptive smoothing factor that comprehensively retains importance, use the adaptive rolling sphere method with the radius dynamically constrained by importance to weave a triangular mesh, and repair closed holes through dynamic programming to generate a non-uniform watertight manifold mesh model.

[0102] This module is configured to receive a discrete simplified point cloud with an extremely spatially non-uniform density output from the third module, and reconstruct it into a three-dimensional manifold watertight mesh with a continuous physical surface through pure mathematical interpolation and spatial weaving algorithms, which is then directly provided to the underlying rendering pipeline for rasterization and depth testing.

[0103] Due to non-uniform folding and specular reflection noise at the sensor's underlying surface, the simplified point cloud surface may exhibit local geometric steps and micro-burrs. Direct triangulation would result in severe normal specular flickering on the rendered training tool surface. Therefore, the system first introduces Moving Least Squares (MLS) for smooth surface projection.

[0104] For any reference point in space The system fits a polynomial equation within its local spherical neighborhood. The system defines a spatial weighting function that controls the influence of surrounding points on the fit. :

[0105] ;

[0106] In traditional MLS algorithms, As a fixed, smooth scaling parameter, but the point cloud density in this system spans several orders of magnitude from the center to the edge. If a fixed... This can lead to catastrophic oversmoothing (loss of detail) in densely populated central regions, while sparsely populated peripheral regions will result in unsolvable fitted equations due to a lack of sufficient points in their neighborhoods. Therefore, the system innovatively... Defined as an inverse proportional adaptive function for the overall importance retention of the third module:

[0107] ;

[0108] in This is the baseline adjustment constant set for the system. Through this strict equation constraint, when the point is in a high-density viewpoint focal point region, The maximum is derived from the minimum. The MLS fit closely follows the original coordinates, preserving fine processing traces; when the point is in a low-density edge area of ​​the field of view, Extremely small, leading to enormous The value allows MLS to capture enough reference points across a huge spatial gap, thereby estimating a smooth and stable extended surface.

[0109] After the point cloud is rigorously smoothed to a two-dimensional continuous manifold, the system initiates an adaptive rolling sphere method to physically weave triangular topological surfaces. The classic rolling sphere method assumes that a virtual sphere of fixed radius rolls on the point set, and the three points touched by the sphere generate a valid Delaunay triangular patch. For non-uniform point clouds, the system must allow the volume of the virtual sphere to dynamically expand and contract in space.

[0110] The system is constructed according to the three-dimensional spatial coordinates. The nonlinear expansion equation of the virtual sphere radius in dynamic evolution :

[0111] ;

[0112] In this equation, the physical definitions of each parameter are extremely rigorous: Based on the minimum contact radius, it is calibrated by the minimum physical detection accuracy of the AR device's sensors; It is strictly defined as the maximum expansion rate constant, which determines the maximum roughness that the mesh can allow in the low-frequency region at the very edge of the field of view; Strictly defined as an importance decay sensitivity factor, it controls the smoothness of the volume expansion transition of the rolling ball as it rolls from the center of the field of vision to the edge. Through this equation, the rolling ball contracts into a "micro-sculptor" at the center of the field of vision, finely weaving a dense network; and expands into a "large brush" at the edge of the field of vision, quickly covering a rough, large surface.

[0113] In physics training, due to the operator's hands or large tools obstructing the sensor's line of sight, the reconstructed mesh inevitably contains topological holes (i.e., boundary edges shared by only one triangular facet). To prevent these holes from being exposed in subsequent rendering and causing clipping, the system extracts the closed polygon boundary sequences of all holes. .

[0114] The system constructs a cost evaluation function based on polygon interior partitioning, which strictly penalizes elongated triangles (maximizing the minimum interior angle) and minimizes the added surface area. The system utilizes dynamic programming algorithms from computer science to construct a... The state transition matrix is ​​solved bottom-up to minimize the cumulative subdivision cost that results in the closure of the entire polygon. Thus, the system mathematically ensures that the final 3D point cloud model output to the GPU is a flawless, absolutely closed, watertight mesh, laying an extremely solid topological foundation for subsequent purely physical and optical hybrid rendering.

[0115] The AR low-latency adaptive hybrid rendering module is configured to receive a non-uniform watertight manifold mesh model and push it into the video memory. It performs variable-rate fragment shading in combination with the absolute gaze center point. It records the real mesh depth value by writing a mask through only opening the depth buffer, so as to perform depth testing and physical-level occlusion culling on the virtual training model. Before output, it uses an asynchronous time warp thread to perform reprojection forward prediction integral.

[0116] This module is configured to receive a non-uniform watertight manifold mesh model with absolute physical closure output from the fourth module, and combine it with the underlying graphics application interface of the AR operating system (such as VulkanAPI or OpenGLES underlying specification) to complete the 3D depth occlusion fusion of the virtual training guidance model and the real physical space under extremely demanding millisecond-level "motion to imaging" latency tolerance, and finally output the light signal to the optical waveguide display module of the AR glasses.

[0117] The continuous triangular mesh data generated in the main memory of the central processing unit (CPU) in the fourth module would incur significant time overhead if transferred via traditional bus copying. Therefore, this system employs a direct memory access (DMA) mechanism. The system directly maps and locks the 3D coordinates of the mesh vertices, normal vectors, and topological index data into vertex buffer objects (VBOs) and index buffer objects (IBOs) using an interleaved memory layout, and pushes them directly into the dedicated video memory (VRAM) of the graphics processing unit (GPU) with the highest bit width.

[0118] During the vertex shader stage of the rendering pipeline, the GPU performs a strict spatial coordinate system transformation for each discrete vertex. Let the homogeneous vertex column vector of the mesh in the local model coordinate system be... The system first multiplies the model matrix by its left matrix. Transform it to the absolute world coordinate system.

[0119] More importantly, because AR glasses use binocular stereoscopic imaging, the left and right eye microdisplays have different physical optical centers and are subject to physical offset based on the user's interpupillary distance (IPD). The system must therefore construct two completely independent view matrices for the main rendering thread. and .

[0120] Meanwhile, given the asymmetric optical properties of the AR glasses lenses, an asymmetric frustum perspective projection matrix was constructed. The system strictly defines projection parameters: Let... The depth of the cutting surface (usually set to the nearest distance of the trainee's hand movement, such as 0.1 meters). The cutting depth is the distance to the cutting surface (e.g., ten meters). These represent the left, right, lower, and upper absolute physical boundaries of the view frustum on the near clipping plane, respectively. The derivation of the system's asymmetric projection matrix is ​​as follows:

[0121] ;

[0122] Discrete vertices through space matrix chain multiplication Then, it is mapped to a four-dimensional homogeneous clipping space, and perspective division is performed in hardware (i.e., The components are simultaneously divided by the homogeneous coordinates. After the components are converted, they are converted into normalized device coordinates (NDC).

[0123] After entering the rasterization and fragment shader stage, in order to release the GPU's pixel fill rate bottleneck, the system directly injects the absolute gaze center point of the first module into the variable rate shader register at the hardware level.

[0124] The system divides the two-dimensional screen space of the AR glasses display into 16x16 pixel tiles. The system calculates the Euclidean distance in screen space between the center of each tile and the projection of the two-dimensional gaze point. For tiles whose distance is less than a preset high-definition threshold (i.e., located in the fovea region of vision), the system issues a 1x1 shading rate to the GPU instruction pool, requiring the GPU to independently calculate the lighting equation and texture sampling for each pixel within the tile, ensuring the ultimate sharpness of the core of vision. For peripheral tiles far from the gaze point, the system dynamically reduces their shading rate to 2x2 or 4x4 (i.e., the shading calculation is performed only once physically for each of the sixteen pixels, and then the color is automatically broadcast by the hardware).

[0125] The most fatal flaw of traditional AR systems is clipping, where virtual models are rendered in front of real objects (such as the back of a hand or a physical wrench) in a way that defies spatial logic, ruining the immersive experience of the training. This system utilizes a non-uniform mesh generated by a pre-module that contains a highly accurate topology of the physical environment to perform "phantom depth masking" technology in the underlying pipeline.

[0126] At the start of rendering each frame, the system first feeds a non-uniform watertight mesh of the real environment into the pipeline. However, in this step, the system forcibly disables the write mask for the color buffer via an API interface, while keeping the write mask for the depth buffer enabled.

[0127] At this point, the GPU will not draw any real-world color pixels on the screen, but it will calculate the depth values ​​of the real mesh with extreme precision. And imprint it in the global depth buffer.

[0128] Subsequently, the system grants color write permissions and begins rendering virtual training indicator models (such as 3D exploded views and virtual screw prompts). When the fragment shader attempts to draw a virtual pixel, the GPU's hardware depth testing unit will assign its depth value... Existing at the same screen coordinates in the Z-Buffer Perform a floating-point comparison. If determined... This means that in the 3D physical world, if the virtual guidance model is behind the real device or the trainee's hand, the GPU hardware will immediately execute a depth culling instruction (Discard), and the virtual pixel will be discarded directly; otherwise, rendering and alpha channel semi-transparent blending are allowed. Through this extremely rigorous depth testing logic, the system achieves flawless, purely physical-level virtual-real occlusion.

[0129] There is a physical delay of tens of milliseconds between sensor sampling and photons hitting the screen. If the trainee turns their head rapidly during this period, the rendered image will lag significantly behind the vestibular sensation, causing severe dizziness. To address this, this system forcibly inserts an asynchronous time-warped, high-frequency, independent thread with the highest operating system privileges at the very end of the rendering pipeline.

[0130] The ATW thread independently polls the IMU registers at frequencies up to one kilohertz. Assume the main rendering thread is... At the start of rendering, the sampled head rotation quaternion is: Once the image is rendered, it will soon be available in... Before submitting the data to the display controller, the ATW thread instantaneously captures the latest angular velocity vector of the IMU. .

[0131] The system utilizes quaternion differential equations and Taylor series expansions to... Latest rotation posture at any moment Perform forward prediction:

[0132] ;

[0133] The system then calculates the attitude deviation matrix. Subsequently, the ATW thread treats the 2D frame buffer that the main thread has just rendered as a 3D texture plane floating in front of it, and utilizes... A rapid inverse geometric affine distortion transformation is performed on the plane. Through this reprojection compensation, the system translates or rotates the head on the screen by the corresponding pixel offset in the last nanosecond before the image is illuminated, thus canceling out the physical micro-movements of the head during rendering and achieving zero-latency visual absolute locking.

[0134] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0135] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0136] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0137] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0138] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A spatial computing point cloud data acquisition and training system based on AR glasses, characterized in that, include: The eye-tracking and gaze vector calculation module is configured to acquire the user's grayscale image of the eyes, extract the pupil edge and fit the geometric center, combine the six-degree-of-freedom spatial pose data of the AR glasses and the Kalman filter, calculate and output the absolute gaze direction vector and the absolute gaze center point in the three-dimensional world coordinate system. The point cloud feature extraction and curvature evaluation module is configured to construct a multi-dimensional spatial binary tree from the original discrete point cloud acquired by the depth sensor to perform local nearest neighbor retrieval, construct the covariance matrix through principal component analysis and solve the eigenvalues, and extract the local surface change rate scalar as the geometric curvature weight. The viewpoint-dependent dynamic mesh shrinking module is configured to receive the absolute gaze direction vector, the absolute gaze center point, and the geometric curvature weights. It calculates the eccentricity angle of each point relative to the gaze center axis and maps it to Gaussian visual importance weights. Then, it combines the depth distance weights to generate a comprehensive retention importance. It uses the fourth-order symmetric basic error quadratic matrix based on importance to evaluate the edge folding cost and perform dynamic downsampling shrinking. The non-uniform topological geometry interpolation reconstruction module is configured to receive the shrunken non-uniform point cloud, construct a moving least squares surface fitting based on the inverse proportional adaptive smoothing factor that comprehensively retains importance, use the adaptive rolling sphere method with the radius dynamically constrained by importance to weave a triangular mesh, and repair closed holes through dynamic programming to generate a non-uniform watertight manifold mesh model. The AR low-latency adaptive hybrid rendering module is configured to receive a non-uniform watertight manifold mesh model and push it into the video memory. It performs variable-rate fragment shading in combination with the absolute gaze center point, and records the real mesh depth value by writing a mask through only opening the depth buffer. This allows for depth testing and physical-level occlusion culling of the virtual training model. Before output, it uses an asynchronous time-warped thread to perform reprojection forward prediction integration.

2. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 1, characterized in that, After acquiring the grayscale image of the eye, the eye-tracking and gaze vector calculation module extracts the set of pixels at the edge of the pupil through edge detection and constructs the conic section algebraic equation. To fit the coordinates of the pupil center, where, and These are the pixel coordinates on the two-dimensional image plane. , , , , , All of these are algebraic coefficients of the algebraic equation of the conic section, and the algebraic coefficients satisfy the discriminant. The constraints are set to ensure that the fitting result is a closed ellipse.

3. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 2, characterized in that, Also includes: The Kalman filter uses a six-dimensional column vector as the state vector, which contains three position coordinate components of the absolute gaze center point in three-dimensional space, and three movement rate components on the corresponding orthogonal spatial axes.

4. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 1, characterized in that, The point cloud feature extraction and curvature evaluation module performs singular value decomposition on the covariance matrix when solving for eigenvalues, obtaining three non-negative eigenvalues ​​arranged in descending order, and then uses the formula... Calculate the local surface change rate scalar; in, The scalar representing the rate of change of the local surface. , , These represent the three non-negative eigenvalues ​​of the covariance matrix after decomposition, arranged in descending order.

5. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 4, characterized in that, The eigenvector corresponding to the minimum eigenvalue is extracted as the surface normal vector of the locally fitted surface, and the inner product constraint formula is enforced. ,in, Represents the surface normal vector. The three-dimensional absolute coordinates representing the absolute viewpoint emission source. It represents the three-dimensional coordinates of local spatial points in the original discrete point cloud, so as to achieve global consistency of the normal field of the entire point cloud.

6. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 1, characterized in that, The viewpoint-dependent dynamic mesh shrinking module solves for the eccentricity angle by using the inverse cosine function between the observation ray vector from the observer to the discrete point and the absolute line-of-sight direction vector, and then substitutes the eccentricity angle into a Gaussian visual importance weighting function. In, among them, Represents discrete points Gaussian visual importance weights Represents discrete points Compared to the eccentricity angle of the line of sight's central axis, This represents the visual attenuation control parameter.

7. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 1, characterized in that, In the viewpoint-dependent dynamic mesh shrinking module: When performing edge folding cost evaluation, the edge folding cost function is: ; in, Represents discrete points and discrete points The folding cost of the connecting edges formed, This represents the new point generated by the folding and is a four-dimensional homogeneous coordinate column vector. The row vector representing the transpose of the new point. and Representing discrete points and discrete points The overall importance of retention and They respectively represent the points assigned to discrete points and discrete points The fourth-order symmetric fundamental error quadratic form matrix, Represents discrete points and discrete points The Euclidean distance between them.

8. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 7, characterized in that, In the non-uniform topological geometry interpolation reconstruction module: The inverse proportional adaptive smoothing factor of the moving least squares surface fitting satisfies the formula ,in Representative point The inverse proportional adaptive smoothing factor at the location, Represents the reference adjustment constant; The adaptive rolling ball method satisfies the equation ,in Represents spatial coordinates The radius of the virtual sphere at that location, Represents the minimum contact radius of the base. Represents the maximum expansion rate constant. Represents spatial coordinates The overall importance of retention at each location This represents a factor that is sensitive to the decay of importance.

9. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 1, characterized in that, When performing depth testing, the AR low-latency adaptive hybrid rendering module: The color write mask for the non-uniform watertight manifold mesh model is turned off while the depth write mask remains on, so that the real mesh depth value can be extracted and stored in the global depth buffer. When the depth value of the virtual training model fragment is greater than the real mesh depth value at the same screen coordinate in the global depth buffer, the hardware performs physical-level occlusion culling.

10. The spatial computing point cloud data acquisition and training system based on AR glasses according to claim 9, characterized in that, Also includes: The asynchronous time-warping thread runs independently of the main rendering thread. Before the rendered image is output, it captures the transient angular velocity vector of the inertial measurement unit and uses Taylor series expansion to predict the latest rotational attitude quaternion. The prediction equation is: ; in The latest rotational attitude quaternion is represented. Represents the transient angular velocity vector. The magnitude of the transient angular velocity vector. It represents the scalar value representing the time difference from the start of rendering to the output of the image. This represents the head rotation quaternion during sampling by the main rendering thread. This represents the quaternion multiplication operator and performs reverse geometric reprojection on the output image based on the predicted attitude deviation.

Images