AR glasses rendering operation method and system based on spatial layering
By employing a spatially layered AR glasses rendering method, the problems of low interaction accuracy of complex 3D models in AR environments, unstable virtual model anchoring, and multimodal interaction timing coordination are solved, achieving an efficient and stable AR model operation experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG MUJIAN NETWORK TECHNOLOGY CO LTD
- Filing Date
- 2026-04-27
- Publication Date
- 2026-05-29
AI Technical Summary
Existing AR glasses suffer from problems such as insufficient semantic understanding of the hierarchical structure of complex 3D models, unstable anchoring of virtual models, lack of coordination of multimodal interaction timing, and contradiction between real-time rendering efficiency and interaction response speed, resulting in low interaction accuracy, high error rate, and poor immersion.
An AR glasses rendering method with spatial layering is adopted. A skeletal hierarchy tree is generated through model preprocessing. Combined with AR scene construction and anchoring, multimodal interaction is realized, and real-time rendering and synchronization are achieved. This includes model preprocessing and spatial layering, AR scene construction and anchoring, multimodal interaction processing, and real-time rendering and synchronization control.
It improves the operability and interaction efficiency of complex models in AR environments, ensures the stability of virtual object positions, lowers the operation threshold, enhances the naturalness and immersion of human-computer interaction, realizes the synchronization of animation, gesture feedback and voice explanation, and provides a high-precision, low-latency AR model operation experience.
Smart Images
Figure CN122115747A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of AR glasses operation, and in particular to an AR glasses rendering operation method and system based on spatial layering. Background Technology
[0002] Augmented reality (AR) glasses, as an important medium for human-computer interaction, have demonstrated great potential in fields such as industrial assembly, education and training, and medical assistance. However, in AR application scenarios involving complex 3D models, existing technologies still face many challenges.
[0003] First, traditional AR systems typically render and interact with the entire 3D model as a single rigid body, lacking semantic understanding of the model's internal hierarchical structure. When users attempt to operate on individual parts of the model (such as pistons in an engine or chips on a circuit board), the system struggles to accurately identify the target area, resulting in low interaction accuracy and a high rate of misoperation.
[0004] Secondly, the anchoring stability of virtual models in physical environments is insufficient. Existing anchoring methods based on planes or point clouds are susceptible to changes in ambient lighting, missing textures, or interference from dynamic objects, causing model drift or jitter, which seriously affects the user's spatial perception and operating experience.
[0005] Furthermore, current AR glasses mostly limit their interaction methods to a single modality, such as supporting only gestures or only voice. Even when some systems attempt to integrate multimodal input, they often lack a unified timing coordination mechanism, resulting in significant delays or asynchrony between gesture feedback, model animations, and voice narration, thus disrupting the immersive experience.
[0006] Furthermore, there is a trade-off between real-time rendering efficiency and interactive response speed. High-polygon meshes of complex models place a heavy rendering burden on mobile devices. Without effective spatial layering and optimization of the model, it is difficult to achieve smooth interactive frame rates while ensuring visual quality.
[0007] In summary, there is a need for an AR glasses rendering operation method and system based on spatial layering to address the shortcomings of existing technologies. Summary of the Invention
[0008] To address the shortcomings of existing technologies, this invention provides an AR glasses rendering method and system based on spatial layering, aiming to solve the aforementioned problems.
[0009] To achieve the above objectives, the present invention provides the following technical solution: an AR glasses rendering operation method based on spatial layering, comprising the following steps:
[0010] Step S1: Model preprocessing and spatial layering, obtain the 3D model data to be displayed; perform bounding box calculation on the 3D model data, and automatically generate a skeletal hierarchy tree reflecting the internal spatial structure of the model based on the hierarchical bounding box algorithm;
[0011] Step S2: AR scene construction and anchoring, real-time scanning and reconstruction of the physical environment, identification and creation of one or more spatial anchor points, and binding the root node of the skeletal hierarchy tree to the spatial anchor points;
[0012] Step S3: Multimodal interaction. The AR glasses continuously capture the user's hand point cloud data through the depth camera. The point cloud data is filtered and denoised, and the spatial coordinates of key points of the hand are extracted as gesture features. The gesture features are matched with a preset gesture template library to identify the user's operation intention. Based on the recognition result, the corresponding operation instructions are executed on the model components associated with the corresponding nodes in the skeletal hierarchy tree.
[0013] Step S4: Real-time rendering and synchronization. Based on the structure of the skeletal hierarchy tree and user interaction commands, the 3D model is rendered in real time to synchronize the timing of model animation, gesture feedback and voice explanation.
[0014] Optionally, the skeletal hierarchy tree in step S1 is obtained in the following way:
[0015] Step A1: Input and initialization: Obtain the 3D model file to be processed, traverse all vertices in the model, calculate the smallest axis-aligned bounding box that can completely wrap the entire model, and use it as the root node of the generated skeletal hierarchy tree.
[0016] Step A2: Recursively construct the BVH tree. Define and implement a recursive function to determine whether the current bounding box node contains only a single indivisible model part. If so, mark it as a leaf node and terminate the recursion. Otherwise, use a heuristic strategy along its longest axis to divide the internal geometry into two groups and continue to construct the subtree.
[0017] Step A3: Output and Binding. Semantize the BVH tree into a skeletal hierarchical tree. Non-leaf nodes serve as logical skeletons that can drive the unified transformation of subordinate components. Leaf nodes correspond to interactive end model components. Simultaneously establish the mapping relationship between each leaf node and its specific model mesh.
[0018] Optionally, step S2 is implemented in the following manner:
[0019] Step B1: Start environmental perception and initialization. After the AR glasses application is launched, its depth camera and IMU sensor are activated, the SLAM system based on AR Foundation is initialized, the physical environment is scanned in real time, and the device pose estimation and 3D scene reconstruction are completed simultaneously.
[0020] Step B2: Spatial feature extraction and anchor point creation. Continuously analyze image frames and depth data to identify stable and textured feature regions. Automatically create one or more spatial anchor points at positions that meet preset stability and scale thresholds for multi-view stable tracking.
[0021] Step B3: Bind the model root node to the spatial anchor point. Set the world transformation matrix of the root node of the skeletal hierarchy tree to be consistent with the pose of the spatial anchor point, so that the position and orientation of the 3D model in physical space are fixed.
[0022] Optionally, step S3 is implemented in the following manner:
[0023] Step C1: Hand point cloud data capture. Using the depth camera built into the AR glasses, the raw point cloud data of the user's hand area is continuously collected. The point cloud stream is subscribed to, and the three-dimensional point set of the local space where the hand is located is obtained in each frame.
[0024] Step C2: Point cloud preprocessing and denoising. Apply distance filtering to remove discrete points that exceed the effective interaction range. Use statistical filtering or radius filtering to remove isolated points caused by sensor noise and retain the dense point cloud of the main hand structure.
[0025] Step C3: Hand key point extraction and feature construction. Input the denoised point cloud into the lightweight hand pose estimation algorithm. The algorithm outputs hand key points and calculates the Euclidean distance, angle or relative pose between key points to form a high-dimensional feature vector.
[0026] Step C4: Gesture recognition and intent matching. The extracted gesture features are matched with the preset gesture template library in real time. The feature matching degree is calculated using algorithms such as dynamic time warping or cosine similarity. When the similarity exceeds the threshold, it is determined to be the corresponding gesture, and the user's operation intent is parsed.
[0027] Step C5: Model component operation and feedback. Based on the identified operation intention, the system traverses the skeletal hierarchy tree, updates the rendering status in real time, and provides visual feedback.
[0028] Optionally, step S3 also includes voice interaction. By integrating the end-side automatic speech recognition module and the emotional text-to-speech module, when a user's voice explanation request is received, the automatic speech recognition module converts the speech into a text command, triggering the emotional text-to-speech module to play a preset explanation audio related to the current model component.
[0029] Optionally, the emotional text-to-speech module can dynamically adjust the tone, speed, and timbre of the narration audio based on the emotional tags in the preset script, providing a more immersive voice narration service.
[0030] Optionally, the gesture template library in step S3 includes at least basic gestures such as single-finger tap, two-finger zoom, and palm drag, which are used to trigger the selection highlight, overall zoom, and spatial translation operations of the model components, respectively.
[0031] Optionally, step S4 is implemented in the following manner:
[0032] Step D1: Layered rendering preparation. Import the automatically generated skeletal hierarchy tree into the Unity3D scene, bind empty GameObjects as logical joints to non-leaf nodes, associate leaf nodes with MeshRenderer, and configure a unified Shader material that supports dynamic highlighting and lighting. At the same time, create a central event scheduler with priority queue and timestamp mechanism outside the Unity main loop.
[0033] Step D2: Unify timing control and state synchronization, process events in the order of timestamps and broadcast synchronization signals to each subsystem, smoothly update skeletal segments at a specified frame rate, trigger the gesture feedback system to instantly modify the material color at the moment the component is selected and restore it when the operation ends;
[0034] Step D3: Frame-level rendering synchronization. All transformation and feedback operations are executed in Unity's LateUpdate() or OnPostRender() stage, and time normalization is performed in conjunction with Time.deltaTime to ensure that the state update is completed before the end of each frame and that the animation speed remains consistent across different device frame rates.
[0035] An AR glasses rendering operating system based on spatial layering, employing the aforementioned AR glasses rendering operation method based on spatial layering, includes a model preprocessing and spatial layering module, an AR scene construction and spatial anchoring module, a multimodal interaction processing module, and a real-time rendering and synchronization control module.
[0036] The model preprocessing and spatial layering module is used to receive the original 3D model data, automatically calculate the axis-aligned bounding box of the model, recursively construct the skeletal hierarchy tree reflecting the internal structure of the model based on the hierarchical bounding box (BVH) algorithm, semanticize BVH into a logical skeletal structure, where non-leaf nodes are driven bones, leaf nodes are bound to specific model parts, and establish the mapping relationship between leaf nodes and model mesh.
[0037] The AR scene construction and spatial anchoring module is used to reconstruct the physical environment in real time using the depth camera and IMU sensor of AR glasses through SLAM technology, identify stable and textured areas in the environment, automatically create spatial anchor points, and bind the root node of the skeletal hierarchy tree to the spatial anchor points, so that the virtual model has a fixed position and orientation in the real space.
[0038] The multimodal interaction processing module includes a gesture recognition submodule and a voice interaction submodule;
[0039] The real-time rendering and synchronization control module is used to render the 3D model in real time according to the structure of the skeletal hierarchy tree and user interaction commands, so as to synchronize the timing of model animation, gesture feedback and voice explanation.
[0040] Optionally, the gesture recognition submodule is used to capture user hand point cloud data, filter and denoise the point cloud, extract key points of the hand and construct a high-dimensional feature vector, match it with a preset gesture template library, identify the operation intention, and trigger the operation on the corresponding node in the skeletal hierarchy tree.
[0041] The voice interaction submodule is used to integrate the on-device automatic speech recognition module, convert user speech into text commands, integrate emotional text into the voice module, and play preset explanatory audio based on the current model component.
[0042] The beneficial effects of this invention are:
[0043] 1. In this invention, by automatically layering the 3D model into a skeletal hierarchical tree according to its internal structure, users can perform precise and intuitive operations on the local components of the model (such as unfolding / folding substructures), which significantly improves the operability and interaction efficiency of complex models in the AR environment. The root node of the model is anchored to the real space, ensuring the positional stability and spatial consistency of the virtual object in the physical world and avoiding drift. Multimodal interaction of gestures and voice is integrated to reduce the operation threshold, improve the naturalness and immersion of human-computer interaction, and achieve strict synchronization of animation, gesture feedback and voice explanation, thereby enhancing the consistency of user experience and the effectiveness of information transmission.
[0044] 2. In this invention, the BVH automatic identification of the internal hierarchical structure of the model is used to achieve intelligent layering of the model without manual annotation, which greatly reduces the preprocessing cost. The semantic skeleton structure enables non-leaf nodes to have logical joint attributes, supporting the driving of complex assemblies in a tree-like manner, which is convenient for hierarchical expansion / collapse. The clear mapping relationship between leaf nodes and mesh provides a precise basis for component selection and highlighting for subsequent interactions. Point cloud filtering effectively suppresses depth camera noise and background interference, improving the accuracy of gesture recognition. Based on high-dimensional feature matching of key points, it supports accurate recognition of complex gestures such as pinching and rotation, which goes beyond simple click and drag. The template matching mechanism makes it easy for the system to expand new gesture commands and improves the flexibility of interaction.
[0045] 3. In this invention, the method is integrated into a single AR system, providing an end-to-end structured model interaction solution. The system architecture supports modular expansion and has good technological foresight and commercial adaptability. The overall system achieves a high-precision, low-latency, and highly immersive AR model operation experience. Through the combination of automated model layering, spatial anchoring, multimodal interaction, and synchronous rendering technologies, it solves the pain points of traditional AR such as the difficulty in fine operation of complex models, unnatural interaction, and fragmented experience. Its beneficial effects are reflected in the four dimensions of ease of use, stability, immersion, and scalability. Attached Figure Description
[0046] Figure 1 This is a schematic diagram of a method flow of the present invention.
[0047] Figure 2 This is a schematic diagram of step S1 of the present invention.
[0048] Figure 3 This is a schematic diagram of step S2 of the present invention.
[0049] Figure 4 This is a schematic diagram of step S3 of the present invention.
[0050] Figure 5 This is a schematic diagram of step S4 of the present invention.
[0051] Figure 6 This is a schematic diagram of a system structure according to the present invention. Detailed Implementation
[0052] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0053] like Figures 1 to 5 As shown, an AR glasses rendering method based on spatial layering includes the following:
[0054] Step S1: Model preprocessing and spatial layering, obtain the 3D model data to be displayed; perform bounding box calculation on the 3D model data, and automatically generate a skeletal hierarchy tree reflecting the internal spatial structure of the model based on the hierarchical bounding box algorithm;
[0055] The skeletal hierarchy tree is obtained in the following way:
[0056] Input and initialization, obtaining raw model data: The system receives the 3D model file to be processed imported by the user, such as .obj or .fbx format. The model is composed of a series of triangular meshes.
[0057] Calculate the global bounding box: Traverse all vertices in the model and calculate the smallest axis-aligned bounding box (AABB) that can completely enclose the entire model. This bounding box will serve as the root node of the skeletal hierarchy tree to be generated.
[0058] Recursively construct the BVH tree, define a recursive partitioning function, and create a recursive function BuildBVH(Node) that takes a bounding box node as input and is responsible for generating child nodes for it.
[0059] To determine the termination condition for the split, the `BuildBVH(Node)` function first checks the geometric complexity of the current node. Specifically, if the current bounding box contains only one independent, indivisible model part, this node is marked as a leaf node, and the recursion on that branch ends.
[0060] Spatial partitioning is performed. If the current node contains multiple components, it needs to be partitioned. The partitioning axis is determined by calculating the length of the current bounding box in the X, Y, and Z dimensions, and the longest axis is selected as the reference axis for this partitioning. This helps to generate a more balanced and efficient tree structure.
[0061] Perform a bisection along the selected longest axis, using a preset heuristic strategy such as surface area heuristic SAH or simple midpoint splitting, to divide all geometry within the current bounding box into two groups, create sub-bounding boxes, and calculate new, smaller AABBs for each of these two groups of geometry.
[0062] Generate child nodes and recursively, using the two newly generated AABBs as the left and right child nodes of the current node. Then, call the BuildBVH(LeftChild) and BuildBVH(RightChild) functions on these two child nodes respectively.
[0063] Output and binding complete the tree structure. Once the recursive process has covered all branches and all of them meet the termination condition, a complete binary BVH tree is constructed. In this context, the tree is given the semantics of a skeletal hierarchical tree: each non-leaf node represents a logical "skeleton" or "joint" used to perform unified spatial transformations on subordinate components, such as rotation and translation; each leaf node directly corresponds to an interactive end component of the model.
[0064] During the construction process, the system records the mapping relationship between each leaf node and the specific model mesh it contains. This allows the system to quickly locate the corresponding leaf node and trace upwards along its parent node chain during subsequent AR interactions when the user selects a component, enabling local or overall animation control.
[0065] The core logic of the recursive function BuildBVH(Node) is expressed as follows:
[0066] The internal logic of the function is a conditional recursive equation:
[0067] ,
[0068] In the formula, the function BuildBVH operates on the geometry set S. If the termination condition is met, a leaf node is generated; otherwise, the set is divided into two parts A and B, generating a node containing left and right subtrees.
[0069] The parameter is defined as follows:
[0070] S is the current set of bounding box nodes, and S is the input received, which is the set of geometry to be processed and its bounding box, containing a list of one or more triangles (mesh) or sub-parts.
[0071] Axis is the dividing axis used to determine which dimension to cut along (X, Y, or Z axis), and the axis with the longest bounding box is selected to ensure efficiency.
[0072] The termination condition is the criterion for determining whether to continue recursion, specifically: TerminateCondition(S) = (|S| ≤ Threshold), where Threshold is the threshold parameter.
[0073] The function BuildBVH includes the following key steps:
[0074] Bounding box calculation formula:
[0075] AABB=MinMax(vertices) means to traverse all vertices in set S, calculate the minimum and maximum coordinates, and generate axis-aligned bounding boxes.
[0076] The partitioning strategy formula is: S→{Sleft,Sright}, which means that the set S is divided into two subsets along the longest axis Axis.
[0077] The midpoint partitioning logic assigns the geometry to the left and right subsets based on the positional relationship between the coordinates of its center point and the midpoint of its bounding box.
[0078] BuildBVH(Node) is called by the parent node to handle complex models containing multiple components.
[0079] The actions are, in sequence, to calculate the current bounding box, select the longest axis, split the geometry, and call the left subtree.
[0080] BuildBVH(LeftChild) recursively processes the left child node, handling the geometry (Slef) on the left side after splitting. The action is to repeat the BuildBVH process on the left set. If the left set still contains multiple parts, the splitting continues; if there is only one part, the process returns to the leaf node.
[0081] BuildBVH(RightChild) is a recursive function for the right child node. It processes the right set of geometry Sright after the split, and performs the same actions as above, recursively processing the right set.
[0082] BuildBVH(Node) is a function called on the parent node, taking the set of geometry or skeletons S contained in the current bounding box node as input. The function first executes the bounding box calculation formula AABB=MinMax(vertices), traversing all vertices in set S to generate the axis-aligned bounding box of the current node. Then, it checks if the number of elements in the set has reached the recursion termination threshold (Threshold). If it has, a leaf node is generated and the function exits. If not, the longest axis of the bounding box is selected as the splitting axis (Axis), and set S is split into a left subset S_left and a right subset S_right according to the midpoint splitting logic. Then, left and right child nodes are created, and BuildBVH(LeftChild) and BuildBVH(RightChild) are called sequentially to recursively build the left and right subtrees.
[0083] BuildBVH(LeftChild) is a recursive function for the left child node, specifically used to process the left-side geometry set S_left after the parent node is split. The function repeatedly executes the complete logic of BuildBVH(Node) on S_left: calculating the bounding box, determining the termination condition, selecting the split axis, splitting at the midpoint, and recursively building the next lower-level subtree. If S_left meets the termination condition, a leaf node is generated; otherwise, recursively splitting continues until all left-side elements have been constructed as leaf nodes, thus completing the construction of the entire left subtree.
[0084] BuildBVH(RightChild) is a recursive function for the right child node, specifically used to process the right-side geometry set S_right after the parent node is split. The function executes the exact same recursive logic as BuildBVH(LeftChild), performing bounding box calculations, axis splitting, recursive drill-down, and leaf node generation on S_left, ultimately completing the construction of the entire right subtree, so that all right-side geometry or skeletons are included in the BVH hierarchy.
[0085] BuildBVH(LeftChild) and BuildBVH(RightChild) are not independently defined new functions, but rather recursive calls to the BuildBVH(Node) function for the left and right subsets. The core logic of the three is completely consistent, all following the conditional recursive equation defined by the formula BuildBVH(S): when TerminateCondition(S) is satisfied, a leaf node is generated; otherwise, the set is split, an intermediate node is generated, and the child node function is recursively called.
[0086] The specific working steps of the three functions are as follows:
[0087] Starting BuildBVH (Node):
[0088] Input: The current node Node, containing a set S of geometry / skeleton.
[0089] Step 1: Calculate the bounding box of the current node, AABB = MinMax(all vertices in S).
[0090] Step 2: Determine the recursion termination condition
[0091] Determine: Is the number of elements in set S less than or equal to Threshold? If yes, mark the current node as a leaf node, end the recursion, and return to the current node; otherwise, proceed to step 3.
[0092] Step 3: Select the dividing axis Axis, rule: take the longest axis of the bounding box (X / Y / Z).
[0093] Step 4: Divide the set by midpoint. Based on the comparison between the center of the geometry and the midpoint of the bounding box, divide S into S_left, S_m, and S_m ...
[0094] Step 5: Create left and right child nodes. Generate a LeftChild node and bind it to S_left. Generate a RightChild node and bind it to S_right.
[0095] Step 6: Recursively build the left subtree by calling BuildBVH(LeftChild).
[0096] Step 7: Recursively build the right subtree by calling BuildBVH(RightChild) and returning the current node (which has already been attached to the left and right subtrees).
[0097] Start BuildBVH(LeftChild)
[0098] Input: Left child node LeftChild, corresponding to the set S_left
[0099] Step a1: Calculate the bounding box of the left child node, AABB_left = MinMax( S_left vertex).
[0100] Step a2: Determine the termination condition: Is the number of elements in S_left ≤ Threshold?
[0101] If yes, mark LeftChild as a leaf node, end the recursion, and return to the left child node; otherwise, proceed to the next step a3.
[0102] Step a3: Select the longest axis of the S_left bounding box;
[0103] Step a4: Divide S_left into S_left_left and S_left_right based on the midpoint.
[0104] Step a5: Create the next level left and right nodes.
[0105] Step a6: Recursively call BuildBVH(left grandchild node).
[0106] Step a7: Recursively call BuildBVH(right grandchild node) to return the completed left subtree node.
[0107] Start BuildBVH(right Child)
[0108] Input: Right child node RightChild, corresponding to the set S_right
[0109] Step b2: Determine the termination condition: Is the number of elements in S_right ≤ Threshold?
[0110] If so, mark RightChild as a leaf node, end the recursion, and return to the right child node; otherwise, proceed to the next step b3.
[0111] Step b3: Select the longest axis of the S_right bounding box.
[0112] Step b4: Divide S_right into S_right_left and S_right_right based on the midpoint.
[0113] Step b5: Create the next level left and right nodes
[0114] Step b6: Recursively call BuildBVH(right grandchild left node)
[0115] Step b7: Recursively call BuildBVH(right grandchild right node) to return the completed right subtree node. Step S2: AR scene construction and anchoring, perform real-time scanning and reconstruction of the physical environment, identify and create one or more spatial anchor points, and bind the root node of the skeletal hierarchy tree to the spatial anchor points;
[0116] Initiating environmental perception and SLAM initialization: After the AR glasses device starts the application, it activates its built-in depth camera and inertial measurement unit (IMU) sensor, initializes the SLAM system based on the AR Foundation framework, Simultaneous Localization and Mapping, and begins to scan the physical environment in real time, simultaneously completing the device's own pose estimation and the reconstruction of the 3D structure of the environment.
[0117] Spatial feature extraction and anchor point creation: The SLAM system continuously analyzes image frames and depth data captured by the camera to identify planes or feature regions in the environment that are rich in texture, stable, and not easily moved, such as desktops, walls, and device bases. When a reliable feature region that meets the preset stability and scale thresholds is detected, the system automatically creates a spatial anchor at that location. If the application scenario involves multiple key interaction areas, such as different operating surfaces of a large device, multiple spatial anchors can be created to achieve stable tracking from multiple perspectives.
[0118] The model root node is bound to the spatial anchor point. The root node of the bone hierarchy tree generated in the previous steps, which is the parent node representing the global coordinates of the entire 3D model, is bound to the selected spatial anchor point in the coordinate system.
[0119] Specifically, the World Transform matrix of the root node is set to be consistent with the pose of the spatial anchor point.
[0120] This binding ensures that the overall position and orientation of the model in physical space remain fixed regardless of how the user moves the AR glasses, avoiding drift or jitter.
[0121] Step S3: Multimodal interaction. The AR glasses continuously capture the user's hand point cloud data through the depth camera. The point cloud data is filtered and denoised, and the spatial coordinates of key points of the hand are extracted as gesture features. The gesture features are matched with a preset gesture template library to identify the user's operation intention. Based on the recognition result, the corresponding operation instructions are executed on the model components associated with the corresponding nodes in the skeletal hierarchy tree.
[0122] Hand point cloud data capture utilizes the depth camera built into the AR glasses, such as ToF or structured light sensing, to continuously collect raw point cloud data of the user's hand area at a frame rate of no less than 30 FPS. The point cloud stream is subscribed to through the AR Foundation's ARPointCloudManager component, and the three-dimensional point set of the local space where the hand is located is obtained in each frame.
[0123] Point cloud preprocessing and denoising: The captured raw point cloud is filtered and denoised. First, distance filtering is applied to remove discrete points that are outside the effective interaction range (e.g., 0.2–1.0 meters). Then, statistical filtering or radius filtering is used to remove isolated points caused by sensor noise, while retaining the dense point cloud of the main hand structure.
[0124] Hand key point extraction and feature construction: The denoised point cloud is input into a lightweight hand pose estimation algorithm, such as MediaPipe Hands or a self-developed CNN model. The algorithm outputs the three-dimensional spatial coordinates (x, y, z) of 21 key points of the hand (including fingertips, knuckles, palm, etc.). The hand skeleton formed by these key points is used as the gesture feature, and the Euclidean distance, angle or relative pose between key points can be further calculated to form a high-dimensional feature vector.
[0125] Gesture recognition and intent matching involves real-time matching of extracted gesture features with a pre-defined gesture template library, which contains at least the following three basic gestures:
[0126] Single-finger click: Extend the index finger and bend the other fingers to select model parts;
[0127] Two-finger zoom: Open / close your thumb and forefinger to zoom the entire model;
[0128] Palm drag: Spread your five fingers and move them to move the entire model or the currently selected part.
[0129] The feature matching degree is calculated using algorithms such as Dynamic Time Warping (DTW) or cosine similarity. When the similarity exceeds a threshold (such as 0.85), it is determined to be the corresponding gesture, and the user's operation intention is parsed.
[0130] For model component operations and feedback, based on the identified operation intent, the system traverses the skeletal hierarchy tree. If it is a click gesture, the system uses raycasting to determine the model component where the user's gaze and hand are pointing, locates its corresponding leaf node, and triggers a highlight or selection state. If it is a zoom or drag gesture, the system applies the corresponding affine transformation, scaling matrix or translation vector, to the root node of the subtree to which the currently selected component belongs or the global root node. The Unity engine updates the model rendering state in real time and provides haptic or visual feedback, such as edge glowing of the component, to complete a complete gesture interaction loop.
[0131] It also includes voice interaction. By integrating an on-device automatic speech recognition module and an emotional text-to-speech module, when a user's voice explanation request is received, the automatic speech recognition module converts the speech into a text command, triggering the emotional text-to-speech module to play a preset explanation audio related to the current model component.
[0132] The emotional text-to-speech module can dynamically adjust the tone, speed, and timbre of the narration audio based on the emotional tags in the preset script, providing a more immersive audio narration service.
[0133] Step S4: Real-time rendering and synchronization. Based on the structure of the skeletal hierarchy tree and user interaction commands, the 3D model is rendered in real time to synchronize the timing of model animation, gesture feedback and voice explanation.
[0134] Based on the layered rendering preparation of the skeletal hierarchy tree, the automatically generated skeletal hierarchy tree is imported into the Unity3D scene. Each non-leaf node is bound to an empty GameObject as a "logical joint", and each leaf node is associated with its corresponding MeshRenderer component. A unified Shader material is configured for all interactive parts, which supports dynamic modification of color and acceptance of spatial lighting at runtime.
[0135] Construct a central event scheduler. Outside of the Unity main loop, create a central event scheduler as the coordination hub for multimodal interactions. This scheduler maintains a priority queue, receives asynchronous events from the gesture recognition module (such as click and drag events) and the voice interaction module (such as the "play explanation" command), and assigns a precise timestamp to each event.
[0136] Unified timing control and state synchronization are implemented. The scheduler processes events in timestamp order and broadcasts synchronization signals to each subsystem.
[0137] After receiving transformation instructions (such as translation and rotation), the model animation system uses coroutines or the AnimationRigging plugin to smoothly interpolate and update the Transform of the corresponding node in the skeletal hierarchy tree at a specified frame rate (such as 60 FPS).
[0138] The gesture feedback system immediately modifies the material color (such as edge glow) of a model part the moment it is selected, and restores it when the operation ends, ensuring that the visual feedback is strictly aligned with the gesture action.
[0139] When a voice narration system is triggered, the scheduler first pauses the currently running non-critical animation (such as automatic rotation), then calls the TTS module to play the audio, while locking the user input to prevent interruption.
[0140] Frame-level rendering is synchronized, and all transformation and feedback operations are executed in Unity's LateUpdate() or OnPostRender() stage to ensure that all state updates are completed before the end of each rendering frame. Unity's Time.deltaTime is used for time normalization to keep the animation speed consistent across different device frame rates.
[0141] like Figure 6 As shown, an AR glasses rendering operating system based on spatial layering adopts the AR glasses rendering operation method based on spatial layering, including a model preprocessing and spatial layering module, an AR scene construction and spatial anchoring module, a multimodal interaction processing module, and a real-time rendering and synchronization control module.
[0142] The model preprocessing and spatial layering module is used to receive the original 3D model data, automatically calculate the axis-aligned bounding box of the model, recursively construct the skeletal hierarchy tree reflecting the internal structure of the model based on the hierarchical bounding box (BVH) algorithm, semanticize BVH into a logical skeletal structure, where non-leaf nodes are driven bones, leaf nodes are bound to specific model parts, and establish the mapping relationship between leaf nodes and model mesh.
[0143] The AR scene construction and spatial anchoring module is used to reconstruct the physical environment in real time using the depth camera and IMU sensor of AR glasses through SLAM technology, identify stable and textured areas in the environment, automatically create spatial anchor points, and bind the root node of the skeletal hierarchy tree to the spatial anchor points, so that the virtual model has a fixed position and orientation in the real space.
[0144] The multimodal interaction processing module includes a gesture recognition submodule and a voice interaction submodule;
[0145] The real-time rendering and synchronization control module is used to render the 3D model in real time according to the structure of the skeletal hierarchy tree and user interaction commands, so as to synchronize the timing of model animation, gesture feedback and voice explanation.
[0146] The gesture recognition submodule is used to capture user hand point cloud data, filter and denoise the point cloud, extract key points of the hand and construct a high-dimensional feature vector, match it with a preset gesture template library, recognize the operation intention, and trigger the operation on the corresponding node in the skeleton hierarchy tree.
[0147] The voice interaction submodule is used to integrate the on-device automatic speech recognition module, convert user speech into text commands, integrate emotional text into the voice module, and play preset explanatory audio based on the current model component.
[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for rendering AR glasses based on spatial layering, characterized in that, Includes the following steps: Step S1: Model preprocessing and spatial layering to obtain the 3D model data to be displayed; The bounding box calculation is performed on the three-dimensional model data, and a skeletal hierarchy tree reflecting the internal spatial structure of the model is automatically generated based on the hierarchical bounding box algorithm. Step S2: AR scene construction and anchoring, real-time scanning and reconstruction of the physical environment, identification and creation of one or more spatial anchor points, and binding the root node of the skeletal hierarchy tree to the spatial anchor points; Step S3: Multimodal interaction. The AR glasses continuously capture the user's hand point cloud data through the depth camera. The point cloud data is filtered and denoised, and the spatial coordinates of key points of the hand are extracted as gesture features. The gesture features are matched with a preset gesture template library to identify the user's operation intention. Based on the recognition result, the corresponding operation instructions are executed on the model components associated with the corresponding nodes in the skeletal hierarchy tree. Step S4: Real-time rendering and synchronization. Based on the structure of the skeletal hierarchy tree and user interaction commands, the 3D model is rendered in real time to synchronize the timing of model animation, gesture feedback and voice explanation.
2. The AR glasses rendering method based on spatial layering according to claim 1, characterized in that, The skeletal hierarchy tree in step S1 is obtained in the following way: Step A1: Input and initialization: Obtain the 3D model file to be processed, traverse all vertices in the model, calculate the smallest axis-aligned bounding box that can completely wrap the entire model, and use it as the root node of the generated skeletal hierarchy tree. Step A2: Recursively construct the BVH tree. Define and implement a recursive function to determine whether the current bounding box node contains only a single indivisible model part. If so, mark it as a leaf node and terminate the recursion. Otherwise, use a heuristic strategy along its longest axis to divide the internal geometry into two groups and continue to construct the subtree. Step A3: Output and Binding. Semantize the BVH tree into a skeletal hierarchical tree. Non-leaf nodes serve as logical skeletons that can drive the unified transformation of subordinate components. Leaf nodes correspond to interactive end model components. Simultaneously establish the mapping relationship between each leaf node and its specific model mesh.
3. The AR glasses rendering method based on spatial layering according to claim 1, characterized in that, Step S2 is implemented in the following manner: Step B1: Start environmental perception and initialization. After the AR glasses application is launched, its depth camera and IMU sensor are activated, the SLAM system based on AR Foundation is initialized, the physical environment is scanned in real time, and the device pose estimation and 3D scene reconstruction are completed simultaneously. Step B2: Spatial feature extraction and anchor point creation. Continuously analyze image frames and depth data to identify stable and textured feature regions. Automatically create one or more spatial anchor points at positions that meet preset stability and scale thresholds for multi-view stable tracking. Step B3: Bind the model root node to the spatial anchor point. Set the world transformation matrix of the root node of the skeletal hierarchy tree to be consistent with the pose of the spatial anchor point, so that the position and orientation of the 3D model in physical space are fixed.
4. The AR glasses rendering method based on spatial layering according to claim 1, characterized in that, Step S3 is implemented in the following manner: Step C1: Hand point cloud data capture. Using the depth camera built into the AR glasses, the raw point cloud data of the user's hand area is continuously collected. The point cloud stream is subscribed to, and the three-dimensional point set of the local space where the hand is located is obtained in each frame. Step C2: Point cloud preprocessing and denoising. Apply distance filtering to remove discrete points that exceed the effective interaction range. Use statistical filtering or radius filtering to remove isolated points caused by sensor noise and retain the dense point cloud of the main hand structure. Step C3: Hand key point extraction and feature construction. Input the denoised point cloud into the lightweight hand pose estimation algorithm. The algorithm outputs hand key points and calculates the Euclidean distance, angle or relative pose between key points to form a high-dimensional feature vector. Step C4: Gesture recognition and intent matching. The extracted gesture features are matched with the preset gesture template library in real time. The feature matching degree is calculated using algorithms such as dynamic time warping or cosine similarity. When the similarity exceeds the threshold, it is determined to be the corresponding gesture, and the user's operation intent is parsed. Step C5: Model component operation and feedback. Based on the identified operation intention, the system traverses the skeletal hierarchy tree, updates the rendering status in real time, and provides visual feedback.
5. The AR glasses rendering method based on spatial layering according to claim 4, characterized in that, Step S3 also includes voice interaction. By integrating the end-side automatic speech recognition module and the emotional text to speech module, when a user's voice explanation request is received, the automatic speech recognition module converts the speech into a text command, triggering the emotional text to speech module to play a preset explanation audio related to the current model component.
6. The AR glasses rendering method based on spatial layering according to claim 5, characterized in that, The emotional text-to-speech module can dynamically adjust the tone, speed, and timbre of the narration audio based on the emotional tags in the preset script, providing a more immersive audio narration service.
7. The AR glasses rendering method based on spatial layering according to claim 1, characterized in that, In step S3, the gesture template library includes at least basic gestures such as single-finger tap, two-finger zoom, and palm drag, which are used to trigger the selection and highlighting, overall zooming, and spatial translation operations of model components, respectively.
8. The AR glasses rendering method based on spatial layering according to claim 1, characterized in that, Step S4 is implemented in the following manner: Step D1: Layered rendering preparation. Import the automatically generated skeletal hierarchy tree into the Unity3D scene, bind empty GameObjects as logical joints to non-leaf nodes, associate leaf nodes with MeshRenderer, and configure a unified Shader material that supports dynamic highlighting and lighting. At the same time, create a central event scheduler with priority queue and timestamp mechanism outside the Unity main loop. Step D2: Unify timing control and state synchronization, process events in the order of timestamps and broadcast synchronization signals to each subsystem, smoothly update skeletal segments at a specified frame rate, trigger the gesture feedback system to instantly modify the material color at the moment the component is selected and restore it when the operation ends; Step D3: Frame-level rendering synchronization. All transformation and feedback operations are executed in Unity's LateUpdate() or OnPostRender() stage, and time normalization is performed in conjunction with Time.deltaTime to ensure that the state update is completed before the end of each frame and that the animation speed remains consistent across different device frame rates.
9. An AR glasses rendering operating system based on spatial layering, employing the AR glasses rendering operation method based on spatial layering as described in any one of claims 1-8, characterized in that, It includes a model preprocessing and spatial layering module, an AR scene construction and spatial anchoring module, a multimodal interaction processing module, and a real-time rendering and synchronization control module; The model preprocessing and spatial layering module is used to receive the original 3D model data, automatically calculate the axis-aligned bounding box of the model, recursively construct the skeletal hierarchy tree reflecting the internal structure of the model based on the hierarchical bounding box (BVH) algorithm, semanticize BVH into a logical skeletal structure, where non-leaf nodes are driven bones, leaf nodes are bound to specific model parts, and establish the mapping relationship between leaf nodes and model mesh. The AR scene construction and spatial anchoring module is used to reconstruct the physical environment in real time using the depth camera and IMU sensor of AR glasses through SLAM technology, identify stable and textured areas in the environment, automatically create spatial anchor points, and bind the root node of the skeletal hierarchy tree to the spatial anchor points, so that the virtual model has a fixed position and orientation in the real space. The multimodal interaction processing module includes a gesture recognition submodule and a voice interaction submodule; The real-time rendering and synchronization control module is used to render the 3D model in real time according to the structure of the skeletal hierarchy tree and user interaction commands, so as to synchronize the timing of model animation, gesture feedback and voice explanation.
10. The AR glasses rendering operating system based on spatial layering according to claim 9, characterized in that, The gesture recognition submodule is used to capture user hand point cloud data, filter and denoise the point cloud, extract key points of the hand and construct a high-dimensional feature vector, match it with a preset gesture template library, identify the operation intention, and trigger the operation on the corresponding node in the skeletal hierarchy tree. The voice interaction submodule is used to integrate the on-device automatic speech recognition module, convert user speech into text commands, integrate emotional text into the voice module, and play preset explanatory audio based on the current model component.