Smart teaching assistance system based on augmented reality

Abstract

The application discloses an intelligent teaching auxiliary system based on augmented reality, which combines an ORB algorithm with a single multi-task deep learning network, simultaneously realizes feature point detection, edge detection and depth estimation tasks, extracts two-dimensional coordinates of key points, edge information and three-dimensional space coordinates, ensures that the positioning of feature points is more accurate and has strong robustness, in addition, by introducing a multi-head attention mechanism, global feature maps and local feature points are weighted and fused, the expression and matching performance of feature points are optimized, the stability and accuracy of feature point detection are further improved, and reliable data input is provided for subsequent camera pose estimation; based on the extracted two-dimensional coordinates, three-dimensional coordinates and camera intrinsic parameters, combined with a PnP algorithm, the pose information of the camera is solved by minimizing the projection error, so that the virtual three-dimensional model can be accurately aligned with the real teaching target object.

Description

Augmented Reality-Based Intelligent Teaching Assistance System Technical Field

[0001] This invention belongs to the field of teaching support systems, and more specifically, relates to an intelligent teaching support system based on augmented reality. Background Technology

[0002] With the development of educational informatization, augmented reality (AR) technology has been gradually introduced into the teaching field. By combining virtual content with real-world scenarios, it provides a completely new way of presenting and interacting with teaching. Unlike traditional flat textbooks and simple multimedia presentations, AR technology can transform abstract theoretical knowledge into visualized three-dimensional virtual models and overlay them onto real objects or scenes, helping students intuitively understand complex concepts and stimulating their interest in learning. For example, in physics teaching, AR can dynamically display the flow path of current in a circuit; in chemistry experiments, molecular structures and reaction processes can be presented to students in a three-dimensional form; in biology teaching, three-dimensional anatomical models of human organs can be compared with real specimens in real time. This combination of virtual and real not only allows students to break through the limitations of time, space, and cognition, but also enables them to deeply understand the teaching content through interactive operations, cultivate their hands-on skills and exploratory spirit, and further enhance the effectiveness and participation of learning.

[0003] Traditional feature point detection algorithms (such as ORB and SIFT) often suffer from inaccurate localization or feature point loss when faced with complex teaching scenarios (such as changes in lighting and partial occlusion), making it difficult to accurately align virtual models with real objects. Furthermore, current deep learning methods typically assign feature point detection, edge detection, and depth estimation tasks to separate networks, which not only increases computational redundancy but also reduces the system's real-time performance, making it difficult to meet the requirements of efficient, real-time feedback in teaching scenarios.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an intelligent teaching assistance system based on augmented reality, which solves the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the basic concept of the technical solution adopted by the present invention is as follows:

[0007] An augmented reality-based intelligent teaching assistance system includes: an image acquisition module for capturing image data of the teaching target object through a camera;

[0008] Multi-task learning feature extraction module: Based on a single multi-task deep learning network, it integrates the ORB algorithm to extract local feature points of the teaching target image, and simultaneously completes edge detection and depth estimation tasks, generating two-dimensional coordinates, edge information and three-dimensional spatial coordinates of the feature points;

[0009] Multi-head attention feature fusion module: used to perform weighted fusion of global feature maps and local feature points using a multi-head attention mechanism to optimize the expression and matching performance of feature points;

[0010] Camera pose estimation module: Combining the two-dimensional and three-dimensional coordinates of feature points output by the multi-task learning module with camera intrinsic parameters, the camera pose information, including the rotation matrix R and translation vector T, is solved by the PnP algorithm.

[0011] Virtual information overlay module: Based on camera pose information, the 3D virtual model is precisely aligned and dynamically overlaid with the real learning scene to achieve virtual-real fusion display;

[0012] Human-computer interaction module: used to support users to interact with virtual 3D models through gesture recognition, voice control or touch operation and to store the data;

[0013] Learning behavior assessment module: Analyzes user learning behavior based on the interaction data to assess learning effectiveness.

[0014] Optionally, the single multi-task deep learning network adopts a design with a shared backbone network and task branch structure. It extracts global feature maps through the shared backbone network and sets up multiple task branches based on the backbone network. The specific structure includes...

[0015] The input layer receives image data of the teaching target object as input to the network.

[0016] A shared backbone network is used to extract global feature maps at multiple scales, which serve as inputs for subsequent task branches.

[0017] The multi-task branch is used to complete three tasks: feature point detection, edge detection, and depth estimation. The task joint optimization module achieves joint optimization of the task branches through the multi-task loss function.

[0018] Optionally, based on a single multi-task deep learning network, the ORB algorithm is integrated to extract local feature points from the image of the teaching target object, and simultaneously complete edge detection and depth estimation tasks, generating the two-dimensional coordinates, edge information, and three-dimensional spatial coordinates of the feature points. The steps are as follows:

[0019] The ORB algorithm is used to extract feature points from the input teaching target image, the FAST algorithm is used to detect the key point positions, and the BRIEF descriptor is used to calculate the description vector of the feature points to obtain the initial two-dimensional coordinates of the feature points. The image is then input into the shared backbone network of a single multi-task deep learning network to extract the global feature map F.

[0020] Based on the global feature map F, the following steps are performed through a multi-task branch network:

[0021] Combining the initial feature point coordinates generated by the ORB algorithm, the final two-dimensional coordinates of the feature points are optimized and output as P = {(x i y i )};

[0022] An edge feature map E is generated through the edge detection branch to help optimize the localization accuracy and robustness of feature points.

[0023] The depth map D of the image is generated through the depth estimation branch, and the two-dimensional coordinates of the final feature points are mapped to three-dimensional space to calculate the three-dimensional coordinates (X). i Y i Z i The specific calculation formula is as follows: Among them, (x i y i D(x) represents the two-dimensional coordinates of the feature point. i y i ) represents the corresponding depth value, f x f y c is the camera focal length. x c y The coordinates of the optical center of the image;

[0024] Through joint optimization of the multi-task loss function The feature point detection, edge detection, and depth estimation are trained with weight balancing, and the specific formula is as follows: Where λ1, λ2, and λ3 are the weighting coefficients of the task loss, respectively.

[0025] Optionally, the steps to optimize the representation and matching performance of feature points by performing weighted fusion of the global feature map and local feature points using a multi-head attention mechanism are as follows:

[0026] Using feature point information P as query Q, and the feature vector of the global feature map F as key K and value V, the formula is used... Perform attention calculations, where Q = W Q P, K = W K F, V = W V F, W Q W K W VLet d be a learnable weight matrix. k The dimension of the key vector;

[0027] The attention mechanism was then extended to a multi-head form, with multiple independent attention branches calculated separately, and the results concatenated and mapped back to the original dimension.

[0028] Optionally, by combining the 2D and 3D coordinates of the feature points output by the multi-task learning module with the camera intrinsic parameters, the camera pose information, including the rotation matrix R and translation vector T, is solved using the PnP algorithm.

[0029] Based on the camera's intrinsic parameter matrix K, the two-dimensional coordinates and three-dimensional coordinates of the feature points are correlated. The intrinsic parameter matrix K is defined as follows: Among them, f x and f y These represent the horizontal and vertical focal lengths of the camera, respectively. x and c y These are the coordinates of the optical center.

[0030] Two-dimensional coordinates (x) i y i ) and the corresponding three-dimensional coordinates (X) i Y i Z i The PnP algorithm is used to solve for the camera's pose information by minimizing the projection error. Its expression is: Finally, the camera pose information, including the rotation matrix R and translation vector T, is output through iterative solution using the PnP algorithm.

[0031] Optional,

[0032] Based on the rotation matrix R and translation vector T, the coordinate system of the virtual 3D model is transformed into the camera coordinate system, and its expression is: P cam =R·P world +T, where P world P represents the point coordinates of the virtual 3D model in the world coordinate system. cam This represents the coordinates of a point in the camera coordinate system after pose transformation;

[0033] The virtual 3D point P in the camera coordinate system is determined by the camera intrinsic parameter matrix K. cam Projected onto a two-dimensional image plane, its expression is: p img =K·P cam .

[0034] The two-dimensional coordinates p obtained from the projection img This renders the virtual 3D model onto the image plane position corresponding to the real scene, so that the virtual content is visually precisely aligned with the real object.

[0035] As the camera pose information changes in real time, the position and viewpoint of the virtual 3D model are dynamically updated to keep it synchronized with the real scene, realizing the dynamic overlay and fusion display of virtual content and real scene. Finally, the final display result is output, completing the precise alignment and virtual-real fusion display of the virtual 3D model and real scene.

[0036] Optionally, the step of analyzing user learning behavior based on the interaction data and evaluating learning effectiveness is as follows:

[0037] Extract feature information related to learning behavior, such as operation frequency, duration, accuracy and operation path. Based on the extracted feature information, analyze the user's learning behavior, including the user's proficiency in operating virtual content, key focus areas and number of incorrect operations.

[0038] By combining the user's interaction with the virtual teaching content, the learning effect of the user can be evaluated to determine whether the user has mastered the relevant knowledge and operation points of the teaching objectives.

[0039] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all of the following advantages at the same time:

[0040] This invention combines the ORB algorithm with a single multi-task deep learning network to simultaneously perform feature point detection, edge detection, and depth estimation tasks. It extracts the 2D coordinates, edge information, and 3D spatial coordinates of key points, ensuring more accurate and robust feature point localization. Furthermore, by introducing a multi-head attention mechanism, it weights and fuses global feature maps and local feature points, optimizing feature point representation and matching performance, further improving the stability and accuracy of feature point detection and providing reliable data input for subsequent camera pose estimation. Based on the extracted 2D and 3D coordinates and camera intrinsic parameters, the PnP algorithm is used to minimize projection error and solve for the camera pose information, thereby ensuring that the virtual 3D model can be accurately aligned with the real teaching target.

[0041] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0042] The accompanying drawings described below are merely some embodiments. Those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0043] In the picture:

[0044] Figure 1 is a block diagram of the teaching support system;

[0045] Figure 2 shows the architecture of a single multi-task deep learning network.

[0046] It should be noted that these accompanying drawings and textual descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art by referring to specific embodiments. Detailed Implementation

[0047] The invention will now be described in further detail with reference to the accompanying drawings.

[0048] Please refer to Figures 1-2. In this embodiment, an intelligent teaching assistance system based on augmented reality is provided, including an image acquisition module for capturing image data of teaching targets through a camera.

[0049] Multi-task learning feature extraction module: Based on a single multi-task deep learning network, it integrates the ORB algorithm to extract local feature points of the teaching target image, and simultaneously completes edge detection and depth estimation tasks, generating two-dimensional coordinates, edge information and three-dimensional spatial coordinates of the feature points;

[0050] Multi-head attention feature fusion module: used to perform weighted fusion of global feature maps and local feature points using a multi-head attention mechanism to optimize the expression and matching performance of feature points;

[0051] Camera pose estimation module: Combining the 2D and 3D coordinates of feature points output by the multi-task learning module with camera intrinsic parameters, the camera pose information, including the rotation matrix, is solved using the PnP algorithm. R Translation vector T ;

[0052] Virtual information overlay module: Based on camera pose information, the 3D virtual model is precisely aligned and dynamically overlaid with the real learning scene to achieve virtual-real fusion display;

[0053] Human-computer interaction module: used to support users to interact with virtual 3D models through gesture recognition, voice control or touch operation and to store the data;

[0054] Learning behavior assessment module: Analyzes user learning behavior based on the interaction data to assess learning effectiveness.

[0055] In this embodiment, the single multi-task deep learning network adopts a shared backbone network and task branch structure design. It extracts global feature maps through the shared backbone network and sets up multiple task branches based on the backbone network. The specific structure includes...

[0056] The input layer receives image data of the teaching target object as input to the network.

[0057] A shared backbone network is used to extract global feature maps at multiple scales, which serve as inputs for subsequent task branches.

[0058] The multi-task branch is used to complete three tasks: feature point detection, edge detection, and depth estimation. The task joint optimization module achieves joint optimization of the task branches through the multi-task loss function.

[0059] In this embodiment, based on a single multi-task deep learning network and incorporating the ORB algorithm, local feature points of the teaching target image are extracted, and edge detection and depth estimation tasks are completed simultaneously. The steps for generating the two-dimensional coordinates, edge information, and three-dimensional spatial coordinates of the feature points are as follows:

[0060] The ORB algorithm is used to extract feature points from the input teaching target image, the FAST algorithm is used to detect the key point positions, and the BRIEF descriptor is used to calculate the description vector of the feature points to obtain the initial two-dimensional coordinates of the feature points. The image is then input into the shared backbone network of a single multi-task deep learning network to extract the global feature map F.

[0061] Based on the global feature map F, the following steps are performed through a multi-task branch network:

[0062] Combining the initial feature point coordinates generated by the ORB algorithm, the final two-dimensional coordinates of the feature points are optimized and output as P = {(x i y i )};

[0063] An edge feature map E is generated through the edge detection branch to help optimize the localization accuracy and robustness of feature points.

[0064] The depth map D of the image is generated through the depth estimation branch, and the two-dimensional coordinates of the final feature points are mapped to three-dimensional space to calculate the three-dimensional coordinates (X). i Y i Z i The specific calculation formula is as follows: Among them, (x i y i D(x) represents the two-dimensional coordinates of the feature point. i y i ) represents the corresponding depth value, f x f y c is the camera focal length. x c y The coordinates of the optical center of the image;

[0065] Through joint optimization of the multi-task loss function The feature point detection, edge detection, and depth estimation are trained with weight balancing, and the specific formula is as follows: Wherein, λ1, λ2, and λ3 are the weight coefficients of the task loss, respectively. In the electrical experiment teaching scenario, students observe real circuit boards through the system's augmented reality function. The system utilizes a single multi-task deep learning network that integrates the ORB algorithm to quickly identify the key locations of components such as resistors, capacitors, and wire interfaces on the circuit board and accurately extract these feature points. At the same time, the system can also detect the edge contours of circuit board components, helping to distinguish the boundaries and connection relationships of different components.

[0066] In this embodiment, the steps for weighted fusion of the global feature map and local feature points using a multi-head attention mechanism to optimize the representation and matching performance of feature points are as follows:

[0067] Using feature point information P as query Q, and the feature vector of the global feature map F as key K and value V, the formula is used... Perform attention calculations, where Q = W Q P, K = W K F, V = W V F, W Q W K W V Let d be a learnable weight matrix. k The dimension of the key vector;

[0068] The attention mechanism was then extended to a multi-head form, with multiple independent attention branches calculated separately, and the results concatenated and mapped back to the original dimension.

[0069] In this embodiment, the camera pose information is solved using the PnP algorithm by combining the two-dimensional and three-dimensional coordinates of the feature points output by the multi-task learning module with the camera intrinsic parameters. This includes the steps of rotating matrix R and translation vector T.

[0070] Based on the camera's intrinsic parameter matrix K, the two-dimensional coordinates and three-dimensional coordinates of the feature points are correlated. The intrinsic parameter matrix K is defined as follows: Among them, f x and f y These represent the horizontal and vertical focal lengths of the camera, respectively. x and c y These are the coordinates of the optical center.

[0071] Two-dimensional coordinates (x) i y i ) and the corresponding three-dimensional coordinates (X) i Y i Z i The PnP algorithm is used to solve for the camera's pose information by minimizing the projection error. Its expression is: Finally, the camera pose information, including the rotation matrix R and translation vector T, is output through iterative solution using the PnP algorithm.

[0072] In this embodiment, the coordinate system of the virtual 3D model is transformed to the camera coordinate system based on the rotation matrix R and translation vector T. The expression for this transformation is: P cam =R·P world +T, where P world P represents the point coordinates of the virtual 3D model in the world coordinate system. cam This represents the coordinates of a point in the camera coordinate system after pose transformation;

[0073] The virtual 3D point P in the camera coordinate system is determined by the camera intrinsic parameter matrix K. cam Projected onto a two-dimensional image plane, its expression is: p img =K·P cam .

[0074] The two-dimensional coordinates p obtained from the projection img This renders the virtual 3D model onto the image plane position corresponding to the real scene, so that the virtual content is visually precisely aligned with the real object.

[0075] As the camera pose information changes in real time, the position and viewpoint of the virtual 3D model are dynamically updated to keep it synchronized with the real scene, realizing the dynamic overlay and fusion display of virtual content and real scene. Finally, the final display result is output, completing the precise alignment and virtual-real fusion display of the virtual 3D model and real scene.

[0076] In electrical experiment teaching, circuit boards are used as teaching targets. Augmented reality (AR) technology is used to overlay virtual three-dimensional models (such as circuit component labels, virtual wiring, dynamic current flow display, etc.) onto real circuit boards to help students understand the circuit connection principle and the function of components.

[0077] During use, the system captures images of the circuit board using a camera and extracts feature points from the board using feature point detection algorithms (such as fusing ORB and deep learning networks). Next, feature matching and coordinate association are performed on multiple feature points on the board (such as resistors, capacitors, and wire interfaces). Then, the coordinates of the points in the virtual 3D model are transformed from the world coordinate system to the camera coordinate system. Subsequently, the 3D points in the camera coordinate system are projected onto a 2D image plane using the camera intrinsic parameter matrix KKK. The system then precisely aligns the virtual resistor and current flow animations with the physical component positions on the real circuit board and renders them on the image plane. When the user moves the camera or changes the viewing angle, the camera's pose information is updated in real time, and the position of the virtual 3D model changes synchronously with the viewpoint, ensuring a stable virtual-real fusion effect. For example, when the camera captures the resistor area on the circuit board, the system overlays virtual resistance value labels (such as "R = 100Ω"). When the user observes the circuit wiring, the virtual current flow animation dynamically displays the direction of current flow in the circuit, helping students intuitively understand the current path.

[0078] In this embodiment, the steps for analyzing user learning behavior based on the interaction data and evaluating learning effectiveness are as follows:

[0079] Extract feature information related to learning behavior, such as operation frequency, duration, accuracy and operation path. Based on the extracted feature information, analyze the user's learning behavior, including the user's proficiency in operating virtual content, key focus areas and number of incorrect operations.

[0080] By combining the user's interaction with the virtual teaching content, the learning effect of the user can be evaluated to determine whether the user has mastered the relevant knowledge and operation points of the teaching objectives.

[0081] For example, in an electrical experiment scenario, the system monitors the number of times the user clicks on various components on the circuit board, the number of times the sliding rheostat is set, and the accuracy and time taken when connecting ammeters and voltmeters, extracting the core characteristic information of the user's operations. For instance, if a user frequently adjusts parameters on an oscilloscope but fails to set them successfully, the system will mark this area as an "error operation area." If a user repeatedly operates on a component on the circuit board but the path is incorrect, the system determines that the user has not yet mastered the relevant knowledge points.

[0082] When a user completes the correct connection path in the ammeter operation task and the number of errors is below a set threshold, the system determines that the user has mastered the knowledge point. However, in the resistor box setting task, if the number of errors is high and the operation takes a long time, the system determines that the user has not yet mastered the resistance adjustment method. If the system detects that the user's error rate in setting the resistor box reaches 70% and the setting time is significantly higher than the average level, the system marks this area as an "unmastered area," prompting the user to practice further. The threshold setting is preset in the auxiliary system according to the actual situation.

[0083] This experiment focuses on electrical experiment teaching scenarios, selecting common electrical experimental equipment (such as circuit boards, oscilloscopes, ammeters, voltmeters, and resistance boxes) as teaching targets. The aim is to verify the performance advantages of this scheme in feature point detection, edge detection, depth estimation, and pose estimation. Through comparative experiments, the accuracy, real-time performance, and robustness of this scheme compared to existing technologies are evaluated.

[0084] 1) Dataset Content

[0085] For electrical experiment teaching, image data containing various electrical experimental equipment was collected. The images cover different lighting conditions, viewing angles, degrees of occlusion, and background complexity to ensure the diversity and challenge of the dataset.

[0086] Teaching objectives:

[0087] Circuit board: Contains the soldering and wiring of various components, such as resistors, capacitors, and transistors. Oscilloscope: A screen and control panel displaying different waveform signals. Voltmeter / Ammeter: Analog and digital display devices with different ranges. Resistor box: Contains adjustment devices with multiple resistance ranges. Ammeter and switching circuit combination: Connection and switching control in complex circuits.

[0088] (2) Image acquisition details: Image resolution: 1920×1080. Number of images: 5000 in total, of which about 1000 were acquired by each type of device.

[0089] Diverse conditions: changes in lighting: strong light, weak light, natural light.

[0090] Perspective changes: front, side, and tilted angle shots. Obstruction: Some equipment components are obstructed by hands, cables, etc. Background complexity: Simple background and lab desktop background.

[0091] (3) The dataset is divided into a training set: 70% (3500 images) – used for network training. A validation set: 15% (750 images) – used for network parameter tuning and performance monitoring. A test set: 15% (750 images) – used for final model performance evaluation.

[0092] Experimental equipment and training setup

[0093] (1) Hardware equipment

[0094] Image acquisition equipment: Industrial-grade high-definition camera (1920×1080 resolution).

[0095] Computing specifications: CPU: Intel Core i9-12900K; GPU: NVIDIA RTX 4090 (24GB VRAM); Memory: 32GB RAM

[0096] Software and Frameworks: Operating System: Ubuntu 20.04; Deep Learning Framework: PyTorch 2.0; Image Processing Libraries: OpenCV, NumPy

[0097] (2) Training parameter settings

[0098] Optimizer: Adam; Learning rate: Initially 0.001, dynamically adjusted using a learning rate decay strategy (ReduceLROnPlateau); Batch size: 16; Number of training epochs: 50; Loss function: Multi-task joint optimization loss function.

[0099] Comparative Example 1: Traditional Feature Extraction Method for a Single Task: The ORB algorithm is used to extract feature points, outputting two-dimensional coordinates and a description vector. The feature points output by the ORB algorithm are directly input into the PnP algorithm to estimate the camera pose information.

[0100] Comparative Example 2: Multi-network independent task learning method: Independent deep learning networks are used to complete the tasks of feature point detection, edge detection and depth estimation, and each task is trained and optimized independently.

[0101]

[0102] The traditional ORB method achieves a feature point detection accuracy of 74.60%, limited by complex lighting and occlusion environments. Independent multi-network schemes improve accuracy to 84.20%, but the lack of ORB algorithm integration results in insufficient robustness in feature point detection. This invention, by fusing the ORB algorithm with a multi-task deep learning network, improves accuracy to 93.50%, significantly optimizing feature point localization and representation performance.

[0103] Traditional methods lack edge detection capabilities. The edge detection accuracy of an independent multi-network scheme is 77.80%. This invention, through feature sharing and task branch optimization, achieves an edge detection accuracy of 89.60%, further improving the localization accuracy of feature points.

[0104] The depth estimation error of the independent multi-network scheme is 3.12 cm, which is somewhat inaccurate. This invention reduces the error to 1.87 cm through joint task optimization and feature sharing, ensuring more accurate depth estimation.

[0105] Traditional ORB methods suffer from significant pose estimation errors, with a rotation error of 2.8° and a translation error of 6.1cm. This invention combines high-precision feature points and 3D coordinates, optimized using the PnP algorithm, reducing the rotation error to 0.9° and the translation error to 2.5cm.

[0106] Traditional ORB methods take 38ms to run, but their performance is limited. Independent multi-network methods offer better performance, but computational redundancy leads to a longer running time (102ms). This invention achieves real-time performance while maintaining accuracy through shared backbone networks and joint task optimization, with a running time of 63ms, meeting the needs of teaching applications.

[0107] This invention is not limited to the embodiments described above. Anyone should understand that structural changes made under the guidance of this invention, and any technical solutions that are the same as or similar to this invention, fall within the protection scope of this invention. Technical aspects, shapes, and structures not described in detail in this invention are all publicly known technologies.

Claims

1. An intelligent teaching assistance system based on augmented reality, characterized in that, include: Image acquisition module: used to capture image data of teaching targets through a camera; Multi-task learning feature extraction module: Based on a single multi-task deep learning network, this network adopts a shared backbone network and task branch structure, integrates the ORB algorithm, extracts local feature points from the image of the teaching target object, and simultaneously completes edge detection and depth estimation tasks, generating the two-dimensional coordinates, edge information, and three-dimensional spatial coordinates of the feature points; Multi-head attention feature fusion module: Used to perform weighted fusion of global feature maps and local feature points using a multi-head attention mechanism to optimize the representation and matching performance of feature points; Camera pose estimation module: Combining the two-dimensional and three-dimensional coordinates of the feature points output by the multi-task learning feature extraction module with camera intrinsic parameters, the camera pose information, including the rotation matrix, is solved using the PnP algorithm. Translation vector Virtual information overlay module: Based on camera pose information, it accurately aligns and dynamically overlays the 3D virtual model with the real learning scene to achieve a virtual-real fusion display; Human-computer interaction module: It supports users to interact with the virtual 3D model and store the data through gesture recognition, voice control or touch operation; Learning Behavior Assessment Module: Analyzes user learning behavior based on interaction data generated by the human-computer interaction module to assess learning effectiveness.

2. The intelligent teaching assistance system based on augmented reality according to claim 1, characterized in that, The single-task multi-task deep learning network adopts a shared backbone network and task branch structure design. The shared backbone network extracts global feature maps, and multiple task branches are set up on the backbone network. The specific structure includes an input layer that receives image data of the teaching target object as network input; a shared backbone network used to extract multi-scale global feature maps, which serve as input to subsequent task branches; multi-task branches, each responsible for completing one of the three tasks: feature point detection, edge detection, and depth estimation; and a joint task optimization module that uses a multi-task loss function to jointly optimize the task branches.

3. The intelligent teaching assistance system based on augmented reality according to claim 1, characterized in that, Based on a single multi-task deep learning network, incorporating the ORB algorithm, this method extracts local feature points from images of teaching target objects and simultaneously performs edge detection and depth estimation tasks, generating the two-dimensional coordinates, edge information, and three-dimensional spatial coordinates of the feature points. The steps are as follows: Feature points are extracted from the input teaching target image using the ORB algorithm; keypoint locations are detected using the FAST algorithm; and description vectors of the feature points are calculated using the BRIEF descriptor to obtain the initial two-dimensional coordinates of the feature points. The image is then input into the shared backbone network of the single multi-task deep learning network to extract the global feature map F. Based on the global feature map F, the following steps are performed through the multi-task branch network: combining the initial feature point coordinates generated by the ORB algorithm, the final two-dimensional coordinates of the feature points are optimized and output. ； An edge feature map E is generated through the edge detection branch to help optimize the localization accuracy and robustness of feature points. The depth map D of the image is generated through the depth estimation branch, and the two-dimensional coordinates of the final feature points are mapped to three-dimensional space to calculate the three-dimensional coordinates. The specific calculation formula is as follows: , in, The two-dimensional coordinates of the feature points For the corresponding depth value, For camera focal length, The coordinates of the optical center of the image are used; a multi-task loss function is jointly optimized. The feature point detection, edge detection, and depth estimation are trained with weight balancing, and the specific formula is as follows: , in, These are the weighting coefficients for the task loss, These represent the loss function values ​​for the feature point detection task, edge detection task, and depth estimation task, respectively.

4. The intelligent teaching assistance system based on augmented reality according to claim 1, characterized in that, The steps for optimizing feature point representation and matching performance by performing weighted fusion of global feature maps and local feature points using a multi-head attention mechanism are as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] As a query Global feature map eigenvectors as keys Sum Through formula Attention calculation is performed, where, The weight matrix is ​​a learnable matrix. Let be the dimension of the key vector; then the attention mechanism is extended to a multi-head form, calculating multiple independent attention branches, and the results are concatenated and mapped back to the original dimension.

5. The intelligent teaching assistance system based on augmented reality according to claim 1, characterized in that, Combining the 2D and 3D coordinates of feature points output by the multi-task learning module with camera intrinsic parameters, the PnP algorithm is used to solve for the camera pose information, including the steps of rotation matrix R and translation vector T. The steps involve associating the 2D and 3D coordinates of the feature points based on the camera's intrinsic parameter matrix K, where K is defined as follows: ,in, and These are the camera's horizontal and vertical focal lengths, respectively. and For the optical center coordinates; convert the two-dimensional coordinates and corresponding three-dimensional coordinates The PnP algorithm is used to solve for the camera's pose information by minimizing the projection error. Its expression is: Finally, the camera pose information, including the rotation matrix, is output through iterative solving using the PnP algorithm. Translation vector ; where λ is a non-zero scale factor, representing the allowed scale relationship between the homogeneous coordinates of a point in three-dimensional space when projected onto the image plane.

6. The intelligent teaching assistance system based on augmented reality according to claim 1, characterized in that, Based on the rotation matrix R and translation vector The expression for transforming the coordinate system of the virtual 3D model to the camera coordinate system is as follows: ,in, This represents the point coordinates of the virtual 3D model in the world coordinate system. This represents the coordinates of a point in the camera coordinate system after pose transformation; the virtual 3D point in the camera coordinate system is represented by the camera intrinsic parameter matrix K. Projected onto a two-dimensional image plane, its expression is: Two-dimensional coordinates obtained from the projection The virtual 3D model is rendered onto the image plane position corresponding to the real scene, so that the virtual content is visually precisely aligned with the real object. As the camera pose information changes in real time, the position and view of the virtual 3D model are dynamically updated to keep it synchronized with the real scene, realizing the dynamic overlay and fusion display of virtual content and real scene. Finally, the final display result is output, completing the precise alignment and virtual-real fusion display of the virtual 3D model and real scene.

7. The intelligent teaching assistance system based on augmented reality according to claim 1, characterized in that, The steps for analyzing user learning behavior and evaluating learning effectiveness based on the interaction data are as follows: extracting feature information related to learning behavior; analyzing user learning behavior based on the extracted feature information; and evaluating user learning effectiveness by combining user interaction with virtual teaching content, and determining whether user has mastered the relevant knowledge and operational points of the teaching target.

Images