Immersive online practical teaching platform based on augmented reality and three-dimensional reconstruction

Abstract

The application discloses an immersive online practical teaching platform based on augmented reality and three-dimensional reconstruction, relates to the technical field of virtual reality, and comprises a semantic-geometry joint modeling module, a cognitive state perception grid distillation module, a skill absorption flux construction module, an adaptive content regulation module, an operation semantic graph construction module and a structured evaluation feedback correction module; the semantic-geometry joint modeling module is used for acquiring multi-view video streams of a target practical scene by using a multi-view image acquisition device and performing space-time alignment processing on the multi-view video streams to obtain aligned scene data; by constructing a semantic-geometry joint implicit field model and a discretized semantic-geometry hybrid grid, the deep fusion of three-dimensional reconstruction of the practical scene and a teaching knowledge graph is realized, the augmented reality teaching view is provided with computable semantic understanding ability, and a continuous multi-dimensional cognitive state vector is generated by fusing eye movement, hand behavior and task context signals.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to an immersive online hands-on teaching platform based on augmented reality and 3D reconstruction. Background Technology

[0002] With the deepening of the digital transformation of vocational education and engineering training, online practical teaching has become an important means to make up for the lack of traditional offline training resources and break through the limitations of time and space. However, most existing online practical teaching systems are still at the stage of two-dimensional video demonstrations or preset three-dimensional animation playback, lacking the ability to reproduce real operation scenarios with high fidelity and the real-time perception and adaptive response mechanism for learners' individual cognitive states.

[0003] On the one hand, traditional 3D reconstruction methods generate scene models that only contain geometric and appearance information, lacking structured teaching semantics such as component functional attributes and assembly logic. This results in a disconnect between teaching content and the 3D scene, failing to support refined teaching intervention based on semantic understanding. On the other hand, existing systems generally adopt a uniform content delivery strategy, failing to dynamically adjust according to learners' real-time cognitive load, skill internalization rate, and operational standardization during practical activities. This easily leads to information overload or cognitive blockage, severely impacting skill acquisition efficiency and immersive learning experience. Furthermore, current practical evaluations largely rely on post-event manual scoring or simple action matching algorithms, lacking the ability to transform the operational process into a structured knowledge representation that combines temporal logic, spatial constraints, and physical feedback. This makes it difficult to achieve fine-grained, interpretable, and personalized feedback correction.

[0004] Therefore, how to construct an immersive online practical teaching platform that integrates semantic-geometric joint modeling, real-time perception of cognitive state, quantitative assessment of skill absorption throughput, and structured evaluation closed-loop feedback has become a key issue that urgently needs to be addressed in the field of intelligent education technology. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides an immersive online hands-on teaching platform based on augmented reality and 3D reconstruction, comprising:

[0008] The module includes a semantic geometry joint modeling module, a cognitive state perception grid distillation module, a skill absorption flux construction module, an adaptive content regulation module, an operational semantic graph construction module, and a structured evaluation feedback correction module.

[0009] The semantic-geometric joint modeling module is used to acquire multi-view video streams of the target practical scene using multi-view image acquisition equipment and perform spatiotemporal alignment processing on the multi-view video streams to obtain aligned scene data. The semantic embedding vector obtained by encoding the pre-set teaching knowledge graph is used as a conditional input. The neural implicit reconstruction network is trained in combination with the aligned scene data to generate a semantic-geometric joint implicit field model containing component semantic label information.

[0010] The cognitive state perception grid distillation module is used to distill discretize the semantic-geometric hybrid grid from the semantic-geometric joint implicit field model, and simultaneously collect eye movement signals, hand operation behavior signals and task context signals during the student's practical task. The eye movement signals, hand operation behavior signals and task context signals are processed by temporal attention fusion to obtain a continuous multidimensional cognitive state vector.

[0011] The skill absorption flux construction module is used to extract the semantic component activation density sequence in the current operating area based on the semantic-geometric hybrid grid, calculate the cross-correlation features between the continuous multidimensional cognitive state vector and the semantic component activation density sequence based on the preset temporal sliding window, and construct a skill absorption flux curve representing the information internalization rate through weighted integral operation based on the cross-correlation features.

[0012] The adaptive content control module is used to calculate the gradient of the skill absorption flux curve to generate augmented reality content control instructions containing parameters for retrospective reinterpretation of historical semantic nodes. Based on the augmented reality content control instructions, it retrieves the associated historical undigested semantic nodes from the semantic-geometric hybrid grid and merges the dimensionality reduction explanation content for the historical undigested semantic nodes with the real-time teaching content to obtain an augmented reality teaching view adapted to the skill internalization rate.

[0013] The operation semantic graph construction module is used to perform boundary detection and segmentation on the continuous hand movement sequence during the student's practical task to obtain the atomic operation unit sequence. The student operation semantic graph is constructed with the atomic operation unit sequence as the node and the temporal sequence relationship and force perception relationship between the atomic operation units as the edge.

[0014] The structured evaluation feedback correction module is used to perform graph matching calculations between the student operation semantic graph and the preset expert standard semantic graph to obtain structured evaluation data, and to feed the structured evaluation data back to the construction process of the skill absorption flux curve to correct the weighted integral calculation weight of the cross-correlation features, so that the subsequently generated augmented reality content control instructions are adapted to the individual skill acquisition characteristics.

[0015] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the training and generation process of the semantic-geometric joint implicit field model specifically includes the following steps:

[0016] The semantic embedding vectors obtained by encoding the component entities, functional attributes and assembly relationships in the pre-set teaching knowledge graph are mapped into conditional feature tensors through a multilayer perceptron, and the conditional feature tensors are concatenated to the spatial features of the intermediate layer of the neural implicit reconstruction network.

[0017] Spatial coordinate points and corresponding observation directions are randomly sampled from the aligned scene data and input into the neural implicit reconstruction network to predict color values, density values, and semantic category probability vectors.

[0018] The photometric loss is calculated based on the actual RGB pixel values ​​and predicted color values ​​in the aligned scene data, and the geometric loss is calculated based on the expected depth obtained by integrating the actual depth values ​​and predicted density values ​​along the ray in the aligned scene data.

[0019] The cross-entropy semantic loss is calculated based on the semantic masks of the components labeled in the aligned scene data and the predicted semantic category probability vectors.

[0020] The total loss function is obtained by weighted summation of photometric loss, geometric loss, and cross-entropy semantic loss. The parameters of the neural implicit reconstruction network are then updated through backpropagation until the total loss function converges, and the semantic-geometric joint implicit field model is output.

[0021] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the distillation generation process of the semantic-geometric hybrid mesh specifically includes the following steps:

[0022] A uniform sampling point array is generated within the bounded three-dimensional space defined by the semantic-geometric joint implicit field model at a preset voxel resolution.

[0023] Traverse each sampling point in the uniform sampling point array, query the density value and semantic category probability vector output by the semantic-geometric joint implicit field model, and filter the sampling points with density values ​​higher than the density threshold and the maximum component in the semantic category probability vector higher than the confidence threshold as the effective vertex set.

[0024] The moving cube algorithm is executed on the set of valid vertices to determine the position of the isosurface based on the density values ​​of adjacent valid vertices and generate a set of triangular facets.

[0025] The semantic category probability vector corresponding to the effective vertex is passed to the three vertices of each triangle in the triangle patch set through barycentric coordinate interpolation, resulting in a semantic geometric hybrid mesh carrying per-vertex semantic labels.

[0026] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the extraction process of the semantic component activation density sequence specifically includes the following steps:

[0027] Construct an augmented reality device's frustum bounding box based on the palm center coordinates and gesture orientation in the student's current hand operation signals;

[0028] Calculate the spatial intersection region between the view frustum bounding box and the semantic-geometry hybrid mesh, and extract all triangular faces located within the spatial intersection region as a subset of candidate faces;

[0029] The number of different semantic labels in the candidate patch subset is counted, and the ratio of the total surface area of ​​the candidate patch subset to the bottom area of ​​the view frustum bounding box is calculated as the patch density factor.

[0030] The original activation intensity value is obtained by multiplying the number of types by the patch density factor. The original activation sequence is obtained by temporally sampling the continuously acquired original activation intensity values ​​at a fixed time step.

[0031] The original activation sequence is normalized to the interval between 0 and 1 using min-maximum normalization, and then high-frequency noise is eliminated by Gaussian smoothing filter to output the semantic component activation density sequence.

[0032] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the specific steps for calculating the cross-correlation features and constructing the skill absorption flux curve are as follows:

[0033] Set the length parameter of the preset timing sliding window Within a preset time-series sliding window, the continuous multidimensional cognitive state vectors are processed respectively. With semantic component activation density sequence Performing mean-reduction processing yields a zero-mean cognitive vector. With zero mean density sequence ;

[0034] Calculate the zero-mean cognitive vector With zero mean density sequence Different time lags Cross-correlation coefficients under ;

[0035] Search for cross-relationships The time lag corresponding to the maximum value As a cognitive response delay;

[0036] With cognitive response delay To realign the continuous multidimensional cognitive state vector and the semantic component activation density sequence by offset, the instantaneous coupling strength sequence is obtained by element-wise multiplying the two aligned sequences. ;

[0037] For instantaneous coupling strength sequence Construct a skill absorption flux curve by performing exponentially decaying weighted integral calculations. .

[0038] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the specific steps of generating augmented reality content control instructions and retrieving historical undigested semantic nodes are as follows:

[0039] Computational skill absorption flux curve The first derivative at the current moment is used as the gradient of change. When the gradient changes When the value is negative and the absolute value exceeds the preset blocking threshold, cognitive blocking is determined and a retrospective reinterpretation mechanism is triggered.

[0040] At the current moment The semantic component activation density sequence is traced back to the endpoint within a preset backtracking time. The timestamps and semantic tags corresponding to the local maxima in the sequence are located, and the located semantic tags are marked as semantic nodes to be backtracked.

[0041] Retrieve a subset of associated facets from a semantic-geometric hybrid mesh that have a direct assembly relationship or functional dependency relationship with the semantic node to be traced back;

[0042] Read the semantic level depth of the associated facet subset, and generate dimensionality-reduced explanatory text and simplified 3D labeled graphics based on the semantic level depth;

[0043] Set the display transparency of the dimensionality reduction explanation text and the simplified 3D labeled graphics to match the changing gradient. The absolute value is directly proportional to the value, and the encapsulation of display transparency, dimensional reduction of text explanation, and simplification of 3D labeled graphics are the commands for adjusting augmented reality content.

[0044] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the boundary detection and segmentation process of the atomic operation unit sequence specifically includes the following steps:

[0045] A continuous sequence of hand movements is input into a pre-trained action segmentation network, which outputs a sequence of probability values ​​for each frame that belong to the operation boundary.

[0046] Non-maximum suppression is performed on the probability value sequence, and local maxima are retained as a set of candidate segmentation points;

[0047] Read the tool switching event timestamp and the abrupt change time point of the force feedback signal from the task context signal, retain the candidate segmentation points in the candidate segmentation point set whose distance from the tool switching event timestamp or the abrupt change time point of the force feedback signal is less than the tolerance threshold, and remove the remaining candidate segmentation points to obtain the verified segmentation point set;

[0048] The continuous hand movement sequence is divided into multiple subsequences based on the time points in the set of verified segmentation points, with each subsequence serving as an atomic operation unit sequence.

[0049] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the specific steps of constructing the student operation semantic graph are as follows:

[0050] For each atomic operation unit sequence, extract the action type label, duration value, and spatial trajectory point cloud, and concatenate and encode the action type label, duration value, and spatial trajectory point cloud into a node vector;

[0051] The time interval between the corresponding node vectors of adjacent atomic operation unit sequences is calculated as the weight value of the temporal edge;

[0052] Calculate the Hausdorff distance between the spatial trajectory point clouds of adjacent atomic operation unit sequences, and use the reciprocal of the Hausdorff distance as the weight value of the spatial neighbor edge.

[0053] Extract the force feedback signal segments corresponding to adjacent atomic operation unit sequences, and calculate the Pearson correlation coefficient between the two force feedback signal segments as the weight value of the force association edge;

[0054] Summarize all node vectors, temporal edges, spatial proximity edges, and force-related edges to construct a semantic graph of student operations.

[0055] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the process of obtaining structured evaluation data through graph matching calculation includes the following steps:

[0056] The graph edit distance algorithm is used to calculate the minimum edit cost path between the student operation semantic graph and the pre-set expert standard semantic graph;

[0057] The operation types in the path with the minimum editing cost are analyzed, and the cumulative cost of node replacement operation is mapped to the technique standardization index, the cumulative cost of edge insertion operation is mapped to the step integrity index, and the cumulative cost of edge deletion operation is mapped to the logical correctness index.

[0058] The indicators of technique standardization, step completeness, and logical correctness are normalized separately, and the three normalized indicators are combined to form structured evaluation data.

[0059] As a preferred embodiment of the immersive online practical teaching platform based on augmented reality and 3D reconstruction described in this invention, the process of correcting the weighted integral calculation of cross-correlation features based on structured evaluation data feedback includes the following steps:

[0060] When the logical correctness index in the structured evaluation data is lower than the preset qualified threshold, the operation conversion link corresponding to the two nodes connected by the deleted edge in the student operation semantic graph is identified. In the process of constructing the skill absorption flux curve, the cross-correlation feature integral weight of the semantic component corresponding to the operation conversion link is increased to the first multiple of the original weight.

[0061] When the standardization index of techniques in the structured evaluation data is lower than the preset qualified threshold, the specific operational actions corresponding to the replaced nodes in the student's operational semantic graph are identified. During the construction of the skill absorption flux curve, the integral weight decay coefficient of the specific operational action within the corresponding time period is applied. Reduce it to a second multiple of the original attenuation coefficient, and extend the cognitive processing evaluation window corresponding to the specific operation action;

[0062] Substitute the revised integral weights and attenuation coefficients into the skill absorption flux curve. The calculation formula enables the generated augmented reality content control instructions to be adapted to the individual's skill acquisition characteristics.

[0063] The beneficial effects of this invention are as follows: By constructing a semantic-geometric joint implicit field model and a discretized semantic-geometric hybrid mesh, a deep integration of 3D reconstruction of practical scenarios and teaching knowledge graphs is achieved, enabling augmented reality teaching views to possess computable semantic understanding capabilities. By fusing eye-tracking, hand-eye coordination, and task context signals to generate continuous multi-dimensional cognitive state vectors, and combining semantic component activation density sequences to construct skill absorption flux curves, the internalization rate of abstract information is transformed into quantifiable and traceable dynamic technical indicators. Based on the changing gradient of the skill absorption flux curve, a backtracking and reinterpretation mechanism for historically undigested semantic nodes is automatically triggered, dynamically adjusting the presentation mode and transparency of augmented reality content, achieving real-time adaptive matching between the pace of teaching supply and individual cognitive load. By performing structured graph matching evaluation between student operation semantic graphs and expert standard semantic graphs, and feeding back the evaluation results to correct the integral weights and decay parameters of the cognitive state assessment model, a closed-loop adaptive teaching mechanism is formed. This effectively solves the problems of rigid content delivery, cognitive blockage and delayed recognition, and insufficient personalized adaptation in traditional immersive teaching, improving the accuracy, fluency, and skill acquisition efficiency of online practical teaching. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1This is a schematic diagram of the system framework of an immersive online hands-on teaching platform based on augmented reality and 3D reconstruction, as described in an embodiment of the present invention. Detailed Implementation

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0068] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0069] Reference Figure 1 As one embodiment of the present invention, this embodiment provides an immersive online practical teaching platform based on augmented reality and 3D reconstruction, comprising:

[0070] The module includes a semantic geometry joint modeling module, a cognitive state perception grid distillation module, a skill absorption flux construction module, an adaptive content regulation module, an operational semantic graph construction module, and a structured evaluation feedback correction module.

[0071] The semantic-geometric joint modeling module is used to acquire multi-view video streams of the target practical scene using multi-view image acquisition equipment and perform spatiotemporal alignment processing on the multi-view video streams to obtain aligned scene data. The semantic embedding vector obtained by encoding the pre-set teaching knowledge graph is used as a conditional input. The neural implicit reconstruction network is trained in combination with the aligned scene data to generate a semantic-geometric joint implicit field model containing component semantic label information.

[0072] Furthermore, the semantic embedding vectors obtained by encoding component entities, functional attributes, and assembly relationships in the pre-set teaching knowledge graph are mapped into conditional feature tensors through a multilayer perceptron, and the conditional feature tensors are concatenated to the spatial features of the intermediate layer of the neural implicit reconstruction network. Spatial coordinate points and corresponding observation directions are randomly sampled from the aligned scene data and input into the neural implicit reconstruction network to predict color values, density values, and semantic category probability vectors.

[0073] The photometric loss is calculated based on the actual RGB pixel values ​​and predicted color values ​​in the aligned scene data; the geometric loss is calculated based on the expected depth obtained by integrating the actual depth values ​​and predicted density values ​​along the ray in the aligned scene data; and the cross-entropy semantic loss is calculated based on the semantic mask of the labeled components in the aligned scene data and the predicted semantic category probability vector.

[0074] Photometric loss The calculation expression is:

[0075] ;

[0076] In the formula, The total number of sampled pixels. The first predicted neural implicit reconstruction network Each pixel's color value To align with the actual RGB pixel values ​​in the scene data, The L1 norm is used instead of the L2 norm because the L2 loss is too sensitive to outlier errors caused by abnormal lighting and occlusion areas, which can easily lead to overly smooth reconstruction results and loss of texture details. The L1 loss has better robustness and can converge stably while maintaining edge sharpness.

[0077] Geometric loss The calculation expression is:

[0078] ;

[0079] In the formula, The total number of sampled rays, It is a function of transmittance. The density value predicted by the network. and These are the proximal and distal boundaries of the ray, respectively. To align the true depth value of the j-th ray in the scene data; this integral form is the depth expectation expression of the volume rendering equation, where the neural implicit field does not explicitly define the surface but implicitly expresses the geometry through the density field. It is necessary to integrate along the ray to convert the continuous density distribution into a scalar depth estimate that can be compared with the true depth. The squared term ensures that the depth error is differentiable and has sufficient penalty for large deviations.

[0080] Cross-entropy semantic loss The calculation expression is:

[0081] ;

[0082] In the formula, The total number of sampling points. The total number of semantic categories. For the first Each sampling point in the category The actual mask label (one-hot encoded) on the surface. The corresponding category in the semantic category probability vector predicted by the network The probability value; the use of cross-entropy instead of mean squared error is because semantic classification is essentially a multi-class probability distribution matching problem. Cross-entropy directly measures the KL divergence between the predicted distribution and the true distribution. In class imbalance scenarios, it provides a more stable gradient signal than mean squared error, avoiding the model from biasing towards the majority class and ignoring the semantic recognition of rare components.

[0083] The total loss function is obtained by weighted summation of photometric loss, geometric loss, and cross-entropy semantic loss. The parameters of the neural implicit reconstruction network are then updated through backpropagation until the total loss function converges, and the semantic-geometric joint implicit field model is output.

[0084] Specifically, the semantic embedding vectors obtained by encoding component entities, functional attributes, and assembly relationships in the pre-set teaching knowledge graph are mapped into conditional feature tensors through a multilayer perceptron. The calculation expression is:

[0085] ;

[0086] In the formula, This represents the original semantic embedding vector output from the knowledge graph encoder. , These are the weight matrices for the first and second layers of the multilayer perceptron, respectively. , For the corresponding bias term, Non-linear activation functions are used to introduce feature interaction capabilities. For the output layer activation function, This represents the conditional feature tensor that matches the spatial feature dimension after mapping. The mapping process is that the original knowledge graph embedding is usually in a discrete symbol space and the dimension is inconsistent with the three-dimensional spatial features. Direct splicing will lead to the obstruction of gradient propagation. By transforming it to a continuous geometric feature space through a multilayer perceptron, the semantic information can be injected into the intermediate layer representation of the reconstruction network in a modulated manner. This allows the network to be subject to the dual constraints of visual observation and prior knowledge structure when predicting color, density and semantic category.

[0087] It should be noted that by mapping the semantic embedding vectors in the teaching knowledge graph to conditional features and concatenating them to the intermediate layer of the reconstruction network, and by combining photometric, geometric, and semantic losses for end-to-end training, the 3D reconstruction process can deeply integrate the functional attributes of components and assembly logic. This ensures that the generated implicit field model maintains high-fidelity visual effects while naturally possessing structured semantic understanding capabilities, providing a computable and semantically consistent unified data foundation for subsequent adaptive teaching based on cognitive states.

[0088] In particular, the design solves the problem of the separation between geometric structure and teaching semantics in traditional 3D reconstruction by deeply integrating the discrete symbolic semantics of knowledge graphs with the continuous spatial representation of neural implicit fields at the feature level. This enables the model to not only restore the appearance of the scene, but also understand the functional dependencies and assembly logic between components. The semantic perception reconstruction mechanism provides unified and differentiable underlying data support for all subsequent teaching interventions based on cognitive states, avoiding cognitive evaluation bias caused by inconsistencies between semantic annotation and geometric model.

[0089] The cognitive state perception grid distillation module is used to distill discretized semantic-geometric hybrid grids from the semantic-geometric joint implicit field model. During the students' practical tasks, eye movement signals, hand operation behavior signals, and task context signals are collected simultaneously. Temporal attention fusion processing is performed on the eye movement signals, hand operation behavior signals, and task context signals to obtain a continuous multidimensional cognitive state vector.

[0090] Furthermore, a uniform sampling point array is generated within the bounded 3D space defined by the semantic-geometric joint implicit field model at a preset voxel resolution; each sampling point in the uniform sampling point array is traversed, and the density value and semantic category probability vector output by the semantic-geometric joint implicit field model are queried. Sampling points with density values ​​higher than the density threshold and the largest component in the semantic category probability vector higher than the confidence threshold are selected as the effective vertex set.

[0091] The moving cube algorithm is executed on the set of valid vertices. The position of the isosurface is determined by interpolation based on the density values ​​of adjacent valid vertices, and a set of triangular patches is generated. The semantic category probability vectors corresponding to the valid vertices are passed to the three vertices of each triangular patch in the set of triangular patches through centroid coordinate interpolation, resulting in a semantic-geometric hybrid mesh carrying per-vertex semantic labels.

[0092] The decision logic for selecting the valid vertex set is based on the joint constraint of the density threshold and the semantic confidence threshold. The calculation expression for the decision condition is as follows:

[0093] ;

[0094] In the formula, It is a uniform sampling point array. Sampling points The density value obtained from the query is... For the preset density threshold, Sampling points Category in semantic category probability vector The amount, To preset the reliability threshold, The set of valid vertices after filtering is defined by the fact that the neural implicit field is defined throughout the entire space but has a clear physical meaning only near the surface of the object. The semantic probability distribution of low-density regions far from the surface tends to be uniform noise. If the moving cube algorithm is executed directly without filtering, a large number of meaningless floating faces will be generated and the semantic label transmission will be polluted. By combining constraints, only sampling points that simultaneously satisfy geometric existence and semantic determinism are retained. This ensures both the topological compactness of the generated mesh and the sufficient discriminativeness of the vertex semantic vectors on which the subsequent barycentric coordinate interpolation depends.

[0095] The expression for transferring the semantic category probability vector corresponding to the effective vertex to the vertices of the triangle through barycentric coordinate interpolation is as follows:

[0096] ;

[0097] In the formula, For the vertex of the target triangle The semantic category probability vector, These are the three original valid vertices of the triangular facet. This is the semantic category probability vector for the corresponding vertex. As vertex Relative to triangle The centroid coordinates and satisfy The use of barycentric coordinate interpolation instead of nearest neighbor assignment is because the new vertices generated by the moving cube algorithm are located between the original valid vertices. Nearest neighbor assignment will cause the semantic labels to appear as blocky discontinuous artifacts on the mesh surface. However, barycentric coordinate interpolation uses the planar linear structure of triangles to achieve a smooth transition of semantic probabilities, so that the distilled hybrid mesh has a continuous and differentiable semantic representation at any position, providing an accurate basis for the subsequent patch-level semantic statistics in the intersection region of view frustums.

[0098] It should be noted that by sampling and selecting high-density, high-confidence vertices at voxel resolution within the three-dimensional space defined by the implicit field, generating triangular patches using the moving cube algorithm, and transmitting semantic labels through centroid coordinate interpolation, an efficient conversion from continuous implicit representation to discrete explicit mesh is achieved. This results in an output hybrid mesh that retains geometric details while carrying per-vertex semantic information, providing a lightweight carrier that balances rendering efficiency and semantic accuracy for real-time interaction and cognitive state perception.

[0099] In particular, the distillation strategy, which combines density and semantic confidence threshold screening with the moving cube algorithm, performs lossy compression of the high-dimensional implicit field while preserving key teaching semantic information. The discretization transformation not only reduces the computational overhead of real-time rendering and interaction, but also ensures the continuous transmission of semantic labels on the grid topology through barycentric coordinate interpolation. This enables the system to maintain sufficient semantic resolution to support refined cognitive state perception and operation area activation analysis while ensuring the smoothness of the teaching view.

[0100] The skill absorption flux construction module is used to extract the semantic component activation density sequence in the current operating region based on the semantic-geometric hybrid grid, calculate the cross-correlation features between the continuous multidimensional cognitive state vector and the semantic component activation density sequence based on the preset temporal sliding window, and construct the skill absorption flux curve representing the information internalization rate through weighted integral operation based on the cross-correlation features.

[0101] Furthermore, based on the palm center coordinates and gesture orientation in the student's current hand operation behavior signal, an augmented reality device's view frustum bounding box is constructed; the spatial intersection region between the view frustum bounding box and the semantic-geometric hybrid mesh is calculated, and all triangular faces located within the spatial intersection region are extracted as candidate face subsets.

[0102] The process of determining the spatial intersection region between the view frustum bounding box and the semantic-geometric hybrid mesh first represents the view frustum bounding box as the intersection of six half-spaces, and then determines whether each triangular facet belongs to a subset of candidate faces. The expression for determining the result is:

[0103] ;

[0104] In the formula, For triangular patch indexes in semantic-geometric hybrid meshes, For dough The coordinates of the center point, and These are the first and second views of the cone-shaped bounding box. The normal vector and offset of each clipping plane The Herveside step function outputs 1 when the independent variable is non-negative and 0 otherwise. For dough A candidate indicator value of 1 indicates that it belongs to a subset of candidate patches. The patch-by-patch clipping determination is based on the fact that the view frustum bounding box changes in real time with the student's hand movement. It is impossible to pre-build a spatial index to accelerate the query. The visibility determination of a single patch can be completed in constant time through analytical half-space testing. Combined with GPU parallel computing, millisecond-level candidate patch extraction is achieved, which meets the real-time frame rate requirements of augmented reality teaching view.

[0105] The number of different semantic labels in the candidate patch subset is counted, and the ratio of the total surface area of ​​the candidate patch subset to the bottom area of ​​the view frustum bounding box is calculated as the patch density factor. The number of types is multiplied by the patch density factor to obtain the original activation intensity value. The original activation intensity value is temporally sampled at a fixed time step to obtain the original activation sequence. The original activation sequence is normalized to the interval between 0 and 1 using minimum and maximum value normalization, and then high-frequency noise is eliminated by Gaussian smoothing filter to output the semantic component activation density sequence.

[0106] The formula for calculating the ratio of the total surface area of ​​the candidate patch subset to the bottom area of ​​the view frustum bounding box is as follows:

[0107] ;

[0108] In the formula, For a subset of candidate facets, For dough area, Let be the projected area of ​​the bounding box of the view frustum on the near-cut plane. The patch density factor is a ratio that is significantly affected by the operating distance. At close range, the physical area covered by the same operation is small but occupies a large proportion of the field of view, while at far range, the opposite is true. Normalizing by dividing by the base area eliminates the interference of distance factors, so that the patch density factor purely reflects the spatial filling degree of semantic content in the current gaze area. Only the original activation intensity value obtained by multiplying by the number of semantic types can remain comparable across different operating distances.

[0109] The expression for calculating the minimum-maximum normalization of the original activation sequence is as follows:

[0110] ;

[0111] In the formula, for The original activation strength value at time 1. To normalize the sliding window length, To prevent small constants with a denominator of zero, The normalized activation intensity value is used. The reason for using a local sliding window instead of a global extremum for normalization is that the dynamic range of activation intensity changes with the task stage during the actual operation. Global normalization will compress the subtle fluctuations in the early low-intensity stage to near zero and lose the ability to distinguish them. Local window normalization adaptively tracks the amplitude distribution of the current stage and ensures that the relative changes in each time period can be completely preserved.

[0112] The output calculation expression for eliminating high-frequency noise using a Gaussian smoothing filter is as follows:

[0113] ;

[0114] In the formula, After normalization The activation intensity value at time. The filter window half-width, The smoothing degree is controlled by the Gaussian kernel standard deviation. The output is the semantic component activation density sequence after filtering. Gaussian smoothing is used instead of simple mean filtering because attention shifts in the practical process have a natural temporal continuity. Mean filtering introduces a rectangular window effect, which causes step artifacts in the activation sequence. The bell-shaped weight distribution of the Gaussian kernel is more in line with the gradual characteristics of human cognitive focus. It can preserve the true temporal contour of the operation intention transition while suppressing high-frequency interference such as sensor jitter and gesture recognition errors.

[0115] It should be noted that by constructing a view frustum bounding box to extract candidate patches within the operating region, and combining the number of semantic categories and patch density factors to calculate the original activation intensity, and then normalizing and Gaussian smoothing to output the activation density sequence, the spatial attention distribution of students in the 3D scene is effectively transformed into a stable and quantifiable temporal signal, eliminating high-frequency noise interference and providing a reliable input basis for accurately assessing the degree of semantic focus in the skill internalization process.

[0116] In particular, the design logic of multiplying the density of patches within the intersection of visual cone spaces with semantic diversity as the activation intensity couples and quantifies the student's visual attention focus with the semantic complexity involved in the current operation. After normalization and smoothing to eliminate sensor noise and instantaneous jitter, a stable temporal signal that can reflect both the operation intention and the cognitive load is obtained. This processing method effectively distinguishes between meaningless gaze wandering and purposeful skill exploration behavior, laying a reliable physiological and behavioral foundation for the accurate calculation of skill absorption flux.

[0117] The adaptive content control module is used to calculate the gradient of the skill absorption flux curve to generate augmented reality content control instructions containing parameters for retrospective reinterpretation of historical semantic nodes. Based on the augmented reality content control instructions, it retrieves the associated historical undigested semantic nodes from the semantic-geometric hybrid grid, and merges the dimensionality-reduced explanation content for the historical undigested semantic nodes with the real-time teaching content to obtain an augmented reality teaching view adapted to the skill internalization rate.

[0118] Furthermore, the length parameter of the preset timing sliding window can be set. Within a preset time-series sliding window, the continuous multidimensional cognitive state vectors are processed respectively. With semantic component activation density sequence Performing mean-reduction processing yields a zero-mean cognitive vector. With zero mean density sequence .

[0119] The calculation expression for performing mean-removal processing on the continuous multidimensional cognitive state vector and the semantic component activation density sequence is as follows:

[0120] ;

[0121] In the formula, and These are the original continuous multidimensional cognitive state vector and the semantic component activation density sequence, respectively. The preset time-series sliding window length, and The mean-free zero-mean sequence is the result of the mean-free operation. The mean-free operation is that each of the two sequences has a non-zero DC component. This component is determined by the individual's baseline cognitive level and the inherent semantic complexity of the scene. It is not related to the real-time cognitive-semantic coupling. If it is not eliminated, it will produce a constant bias term that is not related to the lag in the cross-correlation calculation, which will mask the real time-series causal peak. Only after the mean is removed can the cross-correlation coefficient purely reflect the synchronicity of the dynamic fluctuation part.

[0122] Calculate the zero-mean cognitive vector With zero mean density sequence Different time lags Cross-correlation coefficients under The calculation formula is:

[0123] ;

[0124] Search for cross-relationships The time lag corresponding to the maximum value As a cognitive response delay; with cognitive response delay To realign the continuous multidimensional cognitive state vector and the semantic component activation density sequence by offset, the instantaneous coupling strength sequence is obtained by element-wise multiplying the two aligned sequences. For instantaneous coupling strength sequence Construct a skill absorption flux curve by performing exponentially decaying weighted integral calculations. The calculation formula is:

[0125] ;

[0126] in For the current moment, The attenuation coefficient is the integral result. This represents the value of the skill absorption flux curve at the current moment.

[0127] Computational skill absorption flux curve The first derivative at the current moment is used as the gradient of change. .

[0128] The expression for calculating the gradient is:

[0129] ;

[0130] In the formula, Absorb flux value for the skill at the current moment. This represents the skill absorption flux value at the previous sampling time. To fix the sampling time step, The gradient at the current moment is used; the forward difference approximation of the first derivative is employed because the flux curve itself is already a smoothed signal after exponential decay integration. Analytical differentiation amplifies the numerical instability at the integration boundary, while discrete difference is sufficient to capture the trend reversal of the flux curve while maintaining computational efficiency. When the value is negative and its absolute value exceeds the blocking threshold, it indicates that the student is experiencing continuous difficulty in digesting information, and there is sufficient temporal basis for triggering the backtracking and reinterpretation mechanism.

[0131] When the gradient changes When the value is negative and its absolute value exceeds a preset blocking threshold, cognitive blocking is determined to exist, and a retrospective reinterpretation mechanism is triggered; based on the current moment... The semantic component activation density sequence is traced back to the endpoint within a preset backtracking time. The timestamps and semantic tags corresponding to the local maxima in the sequence are located, and the located semantic tags are marked as semantic nodes to be backtracked.

[0132] Retrieve a subset of associated facets from the semantic-geometric hybrid mesh that have a direct assembly relationship or functional dependency relationship with the semantic node to be traced back; read the semantic level depth of the associated facet subsets, and generate dimensionality-reduced explanatory text and simplified 3D labeled graphics based on the semantic level depth.

[0133] Set the display transparency of the dimensionality reduction explanation text and the simplified 3D labeled graphics to match the changing gradient. The absolute value is directly proportional to the value, and the encapsulation of display transparency, dimensional reduction of text explanation, and simplification of 3D labeled graphics are the commands for adjusting augmented reality content.

[0134] The expression for calculating transparency is:

[0135] ;

[0136] In the formula, To reduce the dimensionality of the content and simplify the display transparency of the 3D annotated graphics, To preset the transparency limit and prevent excessive obscuring of real-time teaching content, It is a proportional scaling factor. The absolute value of the gradient is the change. This proportional mapping means that the deeper the cognitive blockage, the more significant the auxiliary prompts students need to break through the comprehension bottleneck. Transparency, as a direct control parameter of visual salience, has a linear correlation with the blockage intensity, allowing students to intuitively perceive the support provided by the system without additional cognitive processing. At the same time, the upper limit constraint avoids the safety risk of auxiliary content completely obscuring the operation field of vision in extreme blockage situations.

[0137] It should be noted that by performing mean-reduction processing on the cognitive state vector and semantic activation density sequence within a sliding window and calculating the cross-correlation coefficient under different lags to locate the cognitive response delay, and then aligning the sequence accordingly, a skill absorption flux curve is constructed through exponential decay weighted integration. Based on the curve gradient, a backtracking reinterpretation mechanism is triggered, and undigested nodes are retrieved according to semantic association to generate transparent and adaptively reduced-dimensional explanations. This achieves dynamic matching between the pace of teaching content supply and the real-time cognitive load of students, effectively alleviating the interruption of practical operation or comprehension deviation caused by information overload.

[0138] In particular, the underlying consideration for introducing cognitive response delay alignment and exponential decay weighted integral mechanism lies in simulating the temporary storage and integration characteristics of human working memory for practical information. The lag search compensates for the neural transmission delay from perception to cognition, while exponential decay reflects the natural forgetting of unconsolidated skills over time. When the curve gradient drops abnormally, it automatically triggers semantic association-based back-interpretation and adjusts the content transparency according to the degree of obstruction. Bionic cognitive modeling makes the dynamic adjustment of teaching content no longer dependent on preset rules but truly follows the individual's real-time information digestion rhythm, improving the adaptive accuracy of immersive teaching and the naturalness of the learning experience.

[0139] The operation semantic graph construction module is used to perform boundary detection and segmentation on the continuous hand movement sequence during the student's practical task to obtain the atomic operation unit sequence. The student operation semantic graph is constructed with the atomic operation unit sequence as the node and the temporal sequence relationship and force perception relationship between the atomic operation units as the edge.

[0140] Furthermore, the continuous hand motion sequence is input into the pre-trained action segmentation network, which outputs a probability value sequence for each frame belonging to the operation boundary; non-maximum suppression is performed on the probability value sequence, and local maxima points are retained as a set of candidate segmentation points.

[0141] The expression for determining whether to perform non-maximum suppression on the probability value sequence to preserve local maxima is as follows:

[0142] ;

[0143] In the formula, Output of the action segmentation network The probability value that the frame belongs to the operation boundary. The radius of the local window for nonmaximum suppression. The set of candidate segmentation points is used for local maximum screening. The output probability of the action segmentation network usually presents a broad peak rather than a sharp pulse near the real boundary. Direct thresholding will produce multiple adjacent redundant segmentation points. Non-maximum suppression retains only the single point with the highest probability in each local window as the unique boundary candidate, which eliminates duplicate detection and maintains the boundary localization accuracy, providing a clean candidate set for subsequent multimodal secondary verification.

[0144] Read the tool switching event timestamp and the abrupt change time point of the force feedback signal from the task context signal. Keep the candidate segmentation points in the candidate segmentation point set whose distance from the tool switching event timestamp or the abrupt change time point of the force feedback signal is less than the tolerance threshold, and remove the remaining candidate segmentation points to obtain the verified segmentation point set.

[0145] The expression for calculating the distance between the candidate segmentation point and the tool switching event timestamp or the abrupt change time of the force feedback signal is as follows:

[0146] ;

[0147] In the formula, For candidate segmentation point timestamps, Switch the event timestamp for the most recent tool. This is the most recent point in time when force feedback abruptly changed. To preset the tolerance threshold, This is the set of segmented points after verification. The distance verification is because pure kinematic segmentation is prone to producing false boundaries due to short pauses in hand movements or trajectory jitter in complex operation scenarios. Tool switching and force sensation abrupt changes are semantic anchors that cannot be forged at the physical interaction level. The tolerance window allows a certain time deviation to accommodate sensor synchronization errors and individual operation rhythm differences. Candidate points that exceed the window are regarded as kinematic artifacts and are eliminated to ensure that the atomic operation units obtained by segmentation correspond to real functional operation segments.

[0148] The continuous hand movement sequence is divided into multiple subsequences based on the time points in the set of verified segmentation points, with each subsequence serving as an atomic operation unit sequence.

[0149] Furthermore, for each atomic operation unit sequence, action type label, duration value, and spatial trajectory point cloud are extracted, and the action type label, duration value, and spatial trajectory point cloud are concatenated and encoded into a node vector; the time interval between corresponding node vectors of adjacent atomic operation unit sequences is calculated as the weight value of the temporal edge; the Hausdorff distance between the spatial trajectory point clouds of adjacent atomic operation unit sequences is calculated, and the reciprocal of the Hausdorff distance is used as the weight value of the spatial neighbor edge.

[0150] The expression for calculating the Hausdorff distance between the spatial trajectory point clouds of adjacent atomic operation unit sequences is as follows:

[0151] ;

[0152] In the formula, and These represent the set of spatial trajectory point clouds corresponding to two adjacent atomic operation units. and Sets and The point in the middle, Represents Euclidean distance. The Hausdorff distance is used to represent the distance between two point clouds. The Hausdorff distance is chosen over the average distance or DTW because the spatial trajectory of the actual operation may have local offsets but the overall shape is consistent. The average distance is too sensitive to outliers, and although DTW can handle temporal scaling, it ignores the rigid constraints of spatial shape. The Hausdorff distance measures the worst-case matching degree between two point sets, which can effectively capture the essential differences in the spatial configuration of the operation trajectory. Using its reciprocal as the weight of the spatial neighbor edge can make the semantic graph discriminative of the spatial normality of the operation path.

[0153] Extract the force feedback signal segments corresponding to adjacent atomic operation unit sequences, and calculate the Pearson correlation coefficient between the two force feedback signal segments as the weight value of the force association edge.

[0154] The expression for calculating the Pearson correlation coefficient between force feedback signal segments corresponding to adjacent atomic operation unit sequences is as follows:

[0155] ;

[0156] In the formula, and The first and second segments of the stress feedback signal from two adjacent atomic manipulation units are respectively the first and second segments of the stress feedback signal. Each sample value, and This represents the mean of the corresponding segments. The number of sampling points for the segment. The Pearson correlation coefficient is used instead of cross-correlation or cosine similarity because the absolute amplitude of the force signal is significantly affected by non-semantic factors such as grip strength and tool wear. The Pearson coefficient eliminates amplitude offset and scale difference through mean removal and variance normalization, and purely measures the degree of linear correlation between two force signals in the wave pattern. This ensures that the weight of the force correlation edge accurately reflects the tightness of the connection between adjacent operations in the mechanical causal chain rather than accidental force coincidence.

[0157] Summarize all node vectors, temporal edges, spatial proximity edges, and force-related edges to construct a semantic graph of student operations.

[0158] It should be noted that by combining a pre-trained action segmentation network with tool switching events and force feedback mutation points, the continuous hand motion sequence is boundary-checked and segmented to obtain atomic operation units. Then, the action labels, durations, and trajectory point clouds of each unit are extracted and encoded into node vectors. The operation semantic graph is constructed by using time interval, spatial proximity, and force feedback correlation as weights for temporal edges, spatial edges, and force feedback edges, respectively. This transforms unstructured motion data into a structured operation knowledge representation that combines temporal logic, spatial constraints, and physical feedback, laying an interpretable foundation for fine-grained skill evaluation.

[0159] In particular, the design approach of using tool switching events and force feedback abrupt change points to perform secondary verification of action segmentation results fully considers the natural constraint effect of physical interaction events on action boundaries in practical tasks, effectively overcoming the problem of missegmentation in complex operation scenarios caused by pure vision or pure kinematics segmentation. The multi-dimensional weighted edges constructed by time interval, spatial proximity, and force correlation encode the temporal order, spatial coordination, and mechanical causal relationship between atomic operation units into a unified graph structure. The semantic graph representation of multimodal fusion not only preserves the fine-grained details of the operation, but also provides a physically meaningful comparability basis for subsequent structural comparison with expert standard graphs.

[0160] The structured evaluation feedback correction module is used to perform graph matching calculations between the student operation semantic graph and the pre-set expert standard semantic graph to obtain structured evaluation data. The structured evaluation data is then fed back into the construction process of the skill absorption flux curve to correct the weighted integral calculation weight of the cross-correlation features, so that the subsequently generated augmented reality content control instructions are adapted to the individual skill acquisition characteristics.

[0161] Furthermore, a graph edit distance algorithm is used to calculate the minimum edit cost path between the student operation semantic graph and the pre-set expert standard semantic graph.

[0162] The objective function for calculating the minimum edit cost path between the student operation semantic graph and the pre-set expert standard semantic graph using the graph edit distance algorithm is expressed as follows:

[0163]

[0164] In the formula, and These are the student operation semantic graph and the expert standard semantic graph, respectively. The set of all possible node-edge mapping schemes. For a specific mapping scheme, Cost of node replacement The cost of edge replacement, and Let $\mathbf$ be the cost of deleting unmatched nodes and $\mathbf$, respectively. To minimize the editing cost, the optimization problem lies in the fact that the differences between student operations and expert standards may manifest in various forms. The graph edit distance incorporates three types of editing operations—node replacement, edge insertion, and edge deletion—into the same cost function through a unified framework. The process of finding the minimum cost is the process of finding the optimal structure alignment. The resulting editing path naturally carries classification information of the difference type, providing an interpretable structured basis for subsequent mapping into independent evaluation indicators.

[0165] The operation types in the path with the minimum editing cost are analyzed. The cumulative cost of node replacement operation is mapped to the technique standardization index, the cumulative cost of edge insertion operation is mapped to the step integrity index, and the cumulative cost of edge deletion operation is mapped to the logical correctness index. The technique standardization index, step integrity index, and logical correctness index are normalized respectively, and the three normalized indices are combined to form structured evaluation data.

[0166] The calculation expressions for the normalized indicators of technique standardization, step completeness, and logical correctness are as follows:

[0167] ;

[0168] In the formula, For the first The original cumulative cost of each indicator, and These represent the minimum and maximum values ​​of the indicator observed in the historical evaluation sample, respectively. To prevent small constants with a denominator of zero, The normalized index values ​​are as follows: The normalization process is due to the fact that the original cost dimensions of the three types of editing operations are different and the numerical ranges are significantly different. Directly using the original values ​​will lead to one index dominating the comprehensive evaluation. After normalization to a unified range, the three indicators will have an equal weight basis, and the structured evaluation data obtained by combination can reflect the students' true performance in the three dimensions of technique, steps and logic in a balanced way.

[0169] When the logical correctness index in the structured evaluation data is lower than the preset qualified threshold, the operation conversion link corresponding to the two nodes connected by the deleted edge in the student operation semantic graph is identified. In the process of constructing the skill absorption flux curve, the cross-correlation feature integral weight of the semantic component corresponding to the operation conversion link is increased to the first multiple of the original weight.

[0170] The calculation expression for increasing the cross-correlation feature integral weight of the corresponding semantic component when the logical correctness index is lower than the preset qualified threshold is as follows:

[0171] ;

[0172] In the formula, The integral weights of the original cross-correlation features, The first multiple coefficient is greater than 1. The corrected integral weights are used to amplify the logical correctness, which means that students lack key semantic understanding in the operation and transformation process. Increasing the cross-correlation integral weights of the corresponding semantic components in this process is equivalent to increasing the cognitive assessment model's gain on the cognitive-semantic coupling signal in this area. This allows students to clearly see even a slight correct understanding in this process on the skill absorption flux curve, avoiding missing the opportunity for timely intervention due to insufficient assessment sensitivity.

[0173] When the standardization index of techniques in the structured evaluation data is lower than the preset qualified threshold, the specific operational actions corresponding to the replaced nodes in the student's operational semantic graph are identified. During the construction of the skill absorption flux curve, the integral weight decay coefficient of the specific operational action within the corresponding time period is applied. The coefficient is reduced to a second multiple of the original attenuation coefficient, extending the cognitive processing assessment window corresponding to the specific operational action; the corrected integral weight and attenuation coefficient are then substituted into the skill absorption flux curve. The calculation formula enables the generated augmented reality content control instructions to be adapted to the individual's skill acquisition characteristics.

[0174] The calculation expression for reducing the integral weight attenuation coefficient within the corresponding time period when the method standardization index is lower than the preset qualified threshold is as follows:

[0175] ;

[0176] In the formula, The original attenuation coefficient, The coefficient is a second multiple of 1. The reduced attenuation coefficient is the result of the correction. The reduction of this attenuation coefficient is because improper technique often stems from students being forced to move to the next stage before they have completed their cognitive processing of the correct action. Reducing the attenuation rate is equivalent to extending the effective memory window of this operation in the calculation of skill absorption flux. This allows the system to incorporate longer-term historical coupling information when evaluating the cognitive state of this stage, giving students more time to process the correct operational logic behind the improper action. The two correction strategies reshape the cognitive assessment model from the intensity dimension and the time dimension, respectively, so that its parameter configuration can adaptively match the actual ability shortcomings exposed by students.

[0177] It should be noted that the minimum editing cost path between the student's operational semantic graph and the expert standard graph is calculated using the graph edit distance algorithm. The costs of node replacement, edge insertion, and edge deletion are mapped to the indicators of technique standardization, step completeness, and logical correctness, respectively. Then, the cross-correlation integral weights or attenuation coefficients of the corresponding operation links are dynamically adjusted according to the evaluation indicators being below the threshold to correct the construction parameters of the skill absorption flux curve. This forms a closed-loop mechanism in which the evaluation results drive the adaptive optimization of the cognitive assessment model, ensuring that subsequent augmented reality content control instructions always conform to the individual's real skill acquisition trajectory and cognitive processing characteristics.

[0178] Specifically, different editing operation types in the graph edit distance are mapped to three independent evaluation indicators: technique standardization, step completeness, and logical correctness. Based on this, the integral weight and attenuation coefficient in the skill absorption flux construction process are adjusted in reverse. This establishes a direct mapping channel between the evaluation results and the parameters of the cognitive assessment model. This allows the system to dynamically adjust the sensitivity of the cognitive state and the evaluation window length of the specific operation based on the student's ability shortcomings exposed in that operation. The closed-loop feedback mechanism ensures that the control strategy of augmented reality teaching content always revolves around the individual's real skill acquisition bottlenecks, rather than remaining at the level of generalized content push, thus realizing personalized adaptive teaching optimization.

[0179] In summary, this invention achieves deep integration of 3D reconstruction of practical scenarios and teaching knowledge graphs by constructing a semantic-geometric joint implicit field model and a discretized semantic-geometric hybrid mesh. This enables augmented reality teaching views to possess computable semantic understanding capabilities. By fusing eye-tracking, hand-eye coordination, and task context signals to generate continuous multi-dimensional cognitive state vectors, and combining semantic component activation density sequences to construct skill absorption flux curves, the internalization rate of abstract information is transformed into quantifiable and traceable dynamic technical indicators. Based on the changing gradient of the skill absorption flux curve, a backtracking and reinterpretation mechanism for historically undigested semantic nodes is automatically triggered, dynamically adjusting the presentation and transparency of augmented reality content. This achieves real-time adaptive matching between the pace of teaching supply and individual cognitive load. By performing structured graph matching evaluation between student operation semantic graphs and expert standard semantic graphs, and feeding back the evaluation results to correct the integral weights and decay parameters of the cognitive state assessment model, a closed-loop adaptive teaching mechanism is formed. This effectively solves the problems of rigid content delivery, cognitive blockage and delayed recognition, and insufficient personalized adaptation in traditional immersive teaching, improving the accuracy, fluency, and skill acquisition efficiency of online practical teaching.

[0180] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An immersive online hands-on teaching platform based on augmented reality and 3D reconstruction, characterized by: include: The module includes a semantic geometry joint modeling module, a cognitive state perception grid distillation module, a skill absorption flux construction module, an adaptive content regulation module, an operational semantic graph construction module, and a structured evaluation feedback correction module. The semantic-geometric joint modeling module is used to acquire multi-view video streams of the target practical scene using multi-view image acquisition equipment and perform spatiotemporal alignment processing on the multi-view video streams to obtain aligned scene data. The semantic embedding vector obtained by encoding the pre-set teaching knowledge graph is used as a conditional input. The neural implicit reconstruction network is trained in combination with the aligned scene data to generate a semantic-geometric joint implicit field model containing component semantic label information. The cognitive state perception grid distillation module is used to distill discretize the semantic-geometric hybrid grid from the semantic-geometric joint implicit field model, and simultaneously collect eye movement signals, hand operation behavior signals and task context signals during the student's practical task. The eye movement signals, hand operation behavior signals and task context signals are processed by temporal attention fusion to obtain a continuous multidimensional cognitive state vector. The skill absorption flux construction module is used to extract the semantic component activation density sequence in the current operating area based on the semantic-geometric hybrid grid, calculate the cross-correlation features between the continuous multidimensional cognitive state vector and the semantic component activation density sequence based on the preset temporal sliding window, and construct a skill absorption flux curve representing the information internalization rate through weighted integral operation based on the cross-correlation features. The adaptive content control module is used to calculate the gradient of the skill absorption flux curve to generate augmented reality content control instructions containing parameters for retrospective reinterpretation of historical semantic nodes. Based on the augmented reality content control instructions, it retrieves the associated historical undigested semantic nodes from the semantic-geometric hybrid grid and merges the dimensionality reduction explanation content for the historical undigested semantic nodes with the real-time teaching content to obtain an augmented reality teaching view adapted to the skill internalization rate. The operation semantic graph construction module is used to perform boundary detection and segmentation on the continuous hand movement sequence during the student's practical task to obtain the atomic operation unit sequence. The atomic operation unit sequence is used as the node and the temporal sequence and force perception relationship between the atomic operation units are used as the edge to construct the student operation semantic graph. The structured evaluation feedback correction module is used to perform graph matching calculations between the student operation semantic graph and the pre-set expert standard semantic graph to obtain structured evaluation data, and to feed the structured evaluation data back to the construction process of the skill absorption flux curve to correct the weighted integral calculation weight of the cross-correlation features, so that the subsequently generated augmented reality content control instructions are adapted to the individual skill acquisition characteristics.

2. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 1, characterized in that: The training and generation process of the semantic-geometric joint implicit field model includes the following steps: The semantic embedding vectors obtained by encoding the component entities, functional attributes and assembly relationships in the pre-set teaching knowledge graph are mapped into conditional feature tensors through a multilayer perceptron, and the conditional feature tensors are concatenated to the spatial features of the intermediate layer of the neural implicit reconstruction network. Spatial coordinate points and corresponding observation directions are randomly sampled from the aligned scene data and input into the neural implicit reconstruction network to predict color values, density values, and semantic category probability vectors. The photometric loss is calculated based on the actual RGB pixel values ​​and predicted color values ​​in the aligned scene data, and the geometric loss is calculated based on the expected depth obtained by integrating the actual depth values ​​and predicted density values ​​along the ray in the aligned scene data. The cross-entropy semantic loss is calculated based on the semantic masks of the components labeled in the aligned scene data and the predicted semantic category probability vectors. The total loss function is obtained by weighted summation of photometric loss, geometric loss, and cross-entropy semantic loss. The parameters of the neural implicit reconstruction network are then updated through backpropagation until the total loss function converges, and the semantic-geometric joint implicit field model is output.

3. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 2, characterized in that: The distillation generation process of the semantic-geometric hybrid mesh includes the following steps: A uniform sampling point array is generated within the bounded three-dimensional space defined by the semantic-geometric joint implicit field model at a preset voxel resolution. Traverse each sampling point in the uniform sampling point array, query the density value and semantic category probability vector output by the semantic-geometric joint implicit field model, and filter the sampling points with density values ​​higher than the density threshold and the maximum component in the semantic category probability vector higher than the confidence threshold as the effective vertex set. The moving cube algorithm is executed on the set of valid vertices to determine the position of the isosurface based on the density values ​​of adjacent valid vertices and generate a set of triangular facets. The semantic category probability vector corresponding to the effective vertex is passed to the three vertices of each triangle in the triangle patch set through barycentric coordinate interpolation, resulting in a semantic geometric hybrid mesh carrying per-vertex semantic labels.

4. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 3, characterized in that: The extraction process of the semantic component activation density sequence includes the following steps: Construct an augmented reality device's frustum bounding box based on the palm center coordinates and gesture orientation in the student's current hand operation signals; Calculate the spatial intersection region between the view frustum bounding box and the semantic-geometry hybrid mesh, and extract all triangular faces located within the spatial intersection region as a subset of candidate faces; The number of different semantic labels in the candidate patch subset is counted, and the ratio of the total surface area of ​​the candidate patch subset to the bottom area of ​​the view frustum bounding box is calculated as the patch density factor. The original activation intensity value is obtained by multiplying the number of types by the patch density factor. The original activation sequence is obtained by temporally sampling the continuously acquired original activation intensity values ​​at a fixed time step. The original activation sequence is normalized to the interval between 0 and 1 using min-maximum normalization, and then high-frequency noise is eliminated by Gaussian smoothing filter to output the semantic component activation density sequence.

5. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 4, characterized in that: The specific steps for calculating the cross-correlation features and constructing the skill absorption flux curve are as follows: Set the length parameter of the preset timing sliding window Within a preset time-series sliding window, the continuous multidimensional cognitive state vectors are processed respectively. With semantic component activation density sequence Performing mean-reduction processing yields a zero-mean cognitive vector. With zero mean density sequence ; Calculate the zero-mean cognitive vector With zero mean density sequence Different time lags Cross-correlation coefficients under ; Search for cross-relationships The time lag corresponding to the maximum value As a cognitive response delay; With cognitive response delay To realign the continuous multidimensional cognitive state vector and the semantic component activation density sequence by offset, the instantaneous coupling strength sequence is obtained by element-wise multiplying the two aligned sequences. ; For instantaneous coupling strength sequence Construct a skill absorption flux curve by performing exponentially decaying weighted integral calculations. .

6. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 5, characterized in that: The specific steps for generating augmented reality content control instructions and retrieving historical undigested semantic nodes are as follows: Computational skill absorption flux curve The first derivative at the current moment is used as the gradient of change. When the gradient changes When the value is negative and the absolute value exceeds the preset blocking threshold, cognitive blocking is determined and a retrospective reinterpretation mechanism is triggered. At the current moment The semantic component activation density sequence is traced back to the endpoint within a preset backtracking time. The timestamps and semantic tags corresponding to the local maxima in the sequence are located, and the located semantic tags are marked as semantic nodes to be backtracked. Retrieve a subset of associated facets from a semantic-geometric hybrid mesh that have a direct assembly relationship or functional dependency relationship with the semantic node to be traced back; Read the semantic level depth of the associated facet subset, and generate dimensionality-reduced explanatory text and simplified 3D labeled graphics based on the semantic level depth; Set the display transparency of the dimensionality reduction explanation text and the simplified 3D labeled graphics to match the changing gradient. The absolute value is directly proportional to the value, and the encapsulation of display transparency, dimensional reduction of text explanation, and simplification of 3D labeled graphics are the commands for adjusting augmented reality content.

7. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 6, characterized in that: The boundary detection and segmentation process of the atomic operation unit sequence includes the following steps: A continuous sequence of hand movements is input into a pre-trained action segmentation network, which outputs a sequence of probability values ​​for each frame that belong to the operation boundary. Non-maximum suppression is performed on the probability value sequence, and local maxima are retained as a set of candidate segmentation points; Read the tool switching event timestamp and the abrupt change time point of the force feedback signal from the task context signal, retain the candidate segmentation points in the candidate segmentation point set whose distance from the tool switching event timestamp or the abrupt change time point of the force feedback signal is less than the tolerance threshold, and remove the remaining candidate segmentation points to obtain the verified segmentation point set; The continuous hand movement sequence is divided into multiple subsequences based on the time points in the set of verified segmentation points, with each subsequence serving as an atomic operation unit sequence.

8. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 7, characterized in that: The specific steps in constructing the student operation semantic graph are as follows: For each atomic operation unit sequence, extract the action type label, duration value, and spatial trajectory point cloud, and concatenate and encode the action type label, duration value, and spatial trajectory point cloud into a node vector; The time interval between the corresponding node vectors of adjacent atomic operation unit sequences is calculated as the weight value of the temporal edge; Calculate the Hausdorff distance between the spatial trajectory point clouds of adjacent atomic operation unit sequences, and use the reciprocal of the Hausdorff distance as the weight value of the spatial neighbor edge. Extract the force feedback signal segments corresponding to adjacent atomic operation unit sequences, and calculate the Pearson correlation coefficient between the two force feedback signal segments as the weight value of the force association edge; Summarize all node vectors, temporal edges, spatial proximity edges, and force-related edges to construct a semantic graph of student operations.

9. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 8, characterized in that: The process of obtaining structured evaluation data through graph matching calculation includes the following steps: The graph edit distance algorithm is used to calculate the minimum edit cost path between the student operation semantic graph and the pre-set expert standard semantic graph; The operation types in the path with the minimum editing cost are analyzed, and the cumulative cost of node replacement operation is mapped to the technique standardization index, the cumulative cost of edge insertion operation is mapped to the step integrity index, and the cumulative cost of edge deletion operation is mapped to the logical correctness index. The indicators of technique standardization, step completeness, and logical correctness are normalized separately, and the three normalized indicators are combined to form structured evaluation data.

10. The immersive online practical teaching platform based on augmented reality and 3D reconstruction as described in claim 9, characterized in that: The process of correcting the weighted integral of the cross-correlation features based on the structured evaluation data feedback involves the following steps: When the logical correctness index in the structured evaluation data is lower than the preset qualified threshold, the operation conversion link corresponding to the two nodes connected by the deleted edge in the student operation semantic graph is identified. In the process of constructing the skill absorption flux curve, the cross-correlation feature integral weight of the semantic component corresponding to the operation conversion link is increased to the first multiple of the original weight. When the standardization index of techniques in the structured evaluation data is lower than the preset qualified threshold, the specific operational actions corresponding to the replaced nodes in the student's operational semantic graph are identified. During the construction of the skill absorption flux curve, the integral weight decay coefficient of the specific operational action within the corresponding time period is applied. Reduce it to a second multiple of the original attenuation coefficient, and extend the cognitive processing evaluation window corresponding to the specific operation action; Substitute the revised integral weights and attenuation coefficients into the skill absorption flux curve. The calculation formula enables the generated augmented reality content control instructions to be adapted to the individual's skill acquisition characteristics.

Images