Intelligent teaching auxiliary system based on augmented reality

Abstract

The invention discloses an intelligent teaching assistance system based on augmented reality, which combines an ORB algorithm with a single multi-task deep learning network, realizes feature point detection, edge detection and depth estimation tasks at the same time, and extracts two-dimensional coordinates, edge information and three-dimensional space coordinates of key points. In addition, by introducing a multi-head attention mechanism, carrying out weighted fusion on a global feature map and local feature points, and optimizing the expression and matching performance of the feature points, the stability and accuracy of feature point detection are further improved, and the accuracy of feature point detection is improved. And reliable data input is provided for subsequent camera pose estimation. And on the basis of the extracted two-dimensional coordinates, three-dimensional coordinates and internal reference of the camera, minimizing a projection error by combining a PnP algorithm to solve the pose information of the camera, thereby ensuring that the virtual three-dimensional model can be accurately aligned with a real teaching target object.

Description

Technical Field

[0001] The present invention belongs to the field of teaching auxiliary systems, and in particular, relates to an intelligent teaching auxiliary system based on augmented reality. Background Art

[0002] With the development of educational informatization, augmented reality (AR) technology has gradually been introduced into the teaching field. By combining virtual content with real scenes, it provides a new presentation method and interactive experience for teaching. Unlike traditional flat teaching materials and single multimedia presentations, AR technology can transform abstract theoretical knowledge into visual three-dimensional virtual models and superimpose them on real objects or scenes, helping students to intuitively understand complex concepts and stimulate their interest in learning. For example, in physics teaching, AR can dynamically display the flow path of current in the circuit; in chemical experiments, molecular structures and reaction processes can be presented to students in three-dimensional form; in biology teaching, three-dimensional anatomical models of human organs can be compared with real specimens in real time. This combination of virtual and real not only allows students to break through the limitations of time, space and cognition, but also enables them to deeply understand the teaching content through interactive operations, cultivate their hands-on ability and spirit of exploration, and further improve the effect and participation of learning.

[0003] Traditional feature point detection algorithms (such as ORB and SIFT) often have problems with inaccurate positioning or feature point loss when facing complex teaching scenarios (such as lighting changes, partial occlusion, etc.), making it difficult to accurately align virtual models with real objects. In addition, current deep learning methods usually assign feature point detection, edge detection, and depth estimation tasks to independent networks for execution, which not only increases computational redundancy, but also reduces the real-time performance of the system, making it difficult to meet the requirements of teaching scenarios for efficient and real-time feedback.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide an intelligent teaching assistance system based on augmented reality, which solves the problems raised in the above-mentioned background technology.

[0006] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0007] An intelligent teaching auxiliary system based on augmented reality includes: an image acquisition module: used for capturing image data of a teaching target object through a camera;

[0008] Multi-task learning feature extraction module: Based on a single multi-task deep learning network, the ORB algorithm is integrated to extract local feature points of the teaching target image, and simultaneously complete edge detection and depth estimation tasks to generate two-dimensional coordinates, edge information and three-dimensional spatial coordinates of feature points;

[0009] Multi-head attention feature fusion module: used to perform multi-head attention mechanism weighted fusion on global feature maps and local feature points to optimize the expression and matching performance of feature points;

[0010] Camera pose estimation module: Combines the 2D and 3D coordinates of the feature points output by the multi-task learning module with the camera intrinsic parameters, and uses the PnP algorithm to solve the camera pose information, including the rotation matrix R and the translation vector T;

[0011] Virtual information overlay module: Based on the camera posture information, the 3D virtual model is accurately aligned and dynamically overlaid with the real scene to achieve virtual-real fusion display;

[0012] Human-computer interaction module: used to support users to interact with virtual 3D models and store them through gesture recognition, voice control or touch operation;

[0013] Learning behavior evaluation module: analyzes the user's learning behavior based on the interaction data and evaluates the learning effect.

[0014] Optionally, a single multi-task deep learning network adopts a shared backbone network and task branch structure design, extracts global feature maps through a shared backbone network, and sets multiple task branches based on the backbone network. The specific structure includes

[0015] Input layer, the input layer receives the image data of the teaching target as the input of the network;

[0016] Shared backbone network: The shared backbone network is used to extract multi-scale global feature maps as input for subsequent task branches;

[0017] Multi-task branches: The multi-task branches are used to complete the three tasks of feature point detection, edge detection and depth estimation respectively; task joint optimization module: the joint optimization of task branches is realized through the multi-task loss function.

[0018] Optionally, based on a single multi-task deep learning network, the ORB algorithm is integrated to extract local feature points of the teaching target image, and edge detection and depth estimation tasks are completed at the same time. The steps of generating the two-dimensional coordinates, edge information and three-dimensional space coordinates of the feature points are as follows:

[0019] Extract feature points from the input teaching target image using the ORB algorithm, detect key point positions using the FAST algorithm, and calculate description vectors of feature points using the BRIEF descriptor to obtain the two-dimensional coordinates of initial feature points; input the image into a shared backbone network of a single multi-task deep learning network to extract a global feature map F;

[0020] Based on the global feature map F, the following steps are performed by the multi-task branch network:

[0021] Combined with the initial feature point coordinates generated by the ORB algorithm, the two-dimensional coordinates of the final feature point are optimized and outputted. i ,y i )};

[0022] The edge feature map E is generated through the edge detection branch to assist in optimizing the positioning accuracy and robustness of the feature points;

[0023] The depth map D of the image is generated by the depth estimation branch, and the two-dimensional coordinates of the final feature points are mapped to the three-dimensional space to calculate the three-dimensional coordinates (X i , Y i , Z i ), the specific calculation formula is: Among them, (x i ,y i ) is the two-dimensional coordinate of the feature point, D(x i ,y i ) is the corresponding depth value, f x , f y is the focal length of the camera, c x , c y is the coordinate of the optical center of the image;

[0024] Multi-task loss function through joint optimization Weight balance training is performed on feature point detection, edge detection and depth estimation. The specific formula is: Among them, λ 1 ,λ 2 ,λ 3 are the weight coefficients of task loss respectively.

[0025] Optionally, the steps of performing a multi-head attention mechanism weighted fusion on the global feature map and the local feature points to optimize the expression and matching performance of the feature points are as follows:

[0026] The feature point information P is used as the query Q, and the feature vector of the global feature map F is used as the key K and value V. Perform attention calculation, where Q = W Q P, K = W K F、V=W V F.W Q, W K , W V is the learnable weight matrix, d k is the dimension of the key vector;

[0027] The attention mechanism is then expanded to a multi-head form, where multiple independent attention branches are calculated separately, and the results are concatenated and mapped back to the original dimension.

[0028] Optionally, the two-dimensional coordinates and three-dimensional coordinates of the feature points output by the multi-task learning module are combined with the camera internal parameters to solve the camera's pose information through the PnP algorithm, including the steps of rotation matrix R and translation vector T.

[0029] According to the camera's intrinsic parameter matrix K, the two-dimensional coordinates of the feature points are associated with the three-dimensional coordinates. The intrinsic parameter matrix K is defined as: Among them, f x and f y are the horizontal and vertical focal lengths of the camera, c x and c y is the optical center coordinate.

[0030] The two-dimensional coordinates (x i ,y i ) and the corresponding three-dimensional coordinates (X i , Y i , Z i ) Input the PnP algorithm and solve the camera's pose information by minimizing the projection error. The expression is: Finally, the PnP algorithm is used to iteratively solve the camera's position information, including the rotation matrix R and the translation vector T.

[0031] Optional,

[0032] According to the rotation matrix R and the translation vector T, the coordinate system of the virtual three-dimensional model is transformed into the camera coordinate system, and its expression is: cam =R·P world +T, where P world represents the point coordinates of the virtual 3D model in the world coordinate system, P cam Represents the point coordinates in the camera coordinate system after the pose transformation;

[0033] The virtual three-dimensional point P in the camera coordinate system is transformed into cam Projected onto the two-dimensional image plane, its expression is: img =K·P cam .

[0034] According to the two-dimensional coordinate p obtained by the projection img, rendering the virtual 3D model to the image plane position corresponding to the real scene, so that the virtual content is visually accurately aligned with the real object;

[0035] As the camera pose information changes in real time, the position and viewing angle of the virtual 3D model are dynamically updated to keep it synchronized with the real scene, achieving dynamic superposition and fusion display of virtual content and real scene. Finally, the final display result is output to complete the precise alignment and virtual-reality fusion display of the virtual 3D model and the real scene.

[0036] Optionally, the steps of analyzing the user's learning behavior based on the interaction data and evaluating the learning effect are:

[0037] Extract characteristic information related to learning behavior, such as operation frequency, duration, accuracy, and operation path, and analyze the user's learning behavior based on the extracted characteristic information, including the user's proficiency in operating virtual content, key focus areas, and number of incorrect operations;

[0038] Based on the interaction between the user and the virtual teaching content, the user's learning effect is evaluated to determine whether the user has mastered the relevant knowledge and operation points of the teaching target objects.

[0039] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0040] The present invention combines the ORB algorithm with a single multi-task deep learning network, realizes the tasks of feature point detection, edge detection and depth estimation at the same time, extracts the two-dimensional coordinates, edge information and three-dimensional space coordinates of key points, ensures that the positioning of feature points is more accurate and has strong robustness, and in addition, by introducing a multi-head attention mechanism, weighted fusion of global feature maps and local feature points is performed to optimize the expression and matching performance of feature points, further improves the stability and accuracy of feature point detection, and provides reliable data input for subsequent camera pose estimation. Based on the extracted two-dimensional coordinates, three-dimensional coordinates and camera internal parameters, the PnP algorithm is combined to minimize the projection error to solve the camera pose information, thereby ensuring that the virtual three-dimensional model can be accurately aligned with the real teaching target object.

[0041] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0043] In the figure:

[0044] Figure 1 This is the block diagram of the teaching assistance system;

[0045] Figure 2 This is a diagram of a single multi-task deep learning network architecture.

[0046] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] The present invention will now be described in further detail with reference to the accompanying drawings.

[0048] See also Figure 1-2 As shown, in this embodiment, an intelligent teaching auxiliary system based on augmented reality is provided, including an image acquisition module: used to capture image data of a teaching target object through a camera;

[0049] Multi-task learning feature extraction module: Based on a single multi-task deep learning network, the ORB algorithm is integrated to extract local feature points of the teaching target image, and simultaneously complete edge detection and depth estimation tasks to generate two-dimensional coordinates, edge information and three-dimensional spatial coordinates of feature points;

[0050] Multi-head attention feature fusion module: used to perform multi-head attention mechanism weighted fusion on global feature maps and local feature points to optimize the expression and matching performance of feature points;

[0051] Camera pose estimation module: Combines the 2D and 3D coordinates of the feature points output by the multi-task learning module with the camera internal parameters, and uses the PnP algorithm to solve the camera pose information, including the rotation matrix. R and translation vectors T ;

[0052] Virtual information overlay module: Based on the camera posture information, the 3D virtual model is accurately aligned and dynamically overlaid with the real scene to achieve virtual-real fusion display;

[0053] Human-computer interaction module: used to support users to interact with virtual 3D models and store them through gesture recognition, voice control or touch operation;

[0054] Learning behavior evaluation module: analyzes the user's learning behavior based on the interaction data and evaluates the learning effect.

[0055] In this embodiment, the single multi-task deep learning network adopts the design of shared backbone network and task branch structure, extracts global feature map through shared backbone network, and sets multiple task branches on the basis of backbone network. The specific structure includes

[0056] Input layer, the input layer receives the image data of the teaching target as the input of the network;

[0057] Shared backbone network: The shared backbone network is used to extract multi-scale global feature maps as input for subsequent task branches;

[0058] Multi-task branches: The multi-task branches are used to complete the three tasks of feature point detection, edge detection and depth estimation respectively; task joint optimization module: the joint optimization of task branches is realized through the multi-task loss function.

[0059] In this embodiment, based on a single multi-task deep learning network, the ORB algorithm is integrated to extract local feature points of the teaching target image, and edge detection and depth estimation tasks are completed at the same time. The steps of generating the two-dimensional coordinates, edge information and three-dimensional space coordinates of the feature points are as follows:

[0060] Extract feature points from the input teaching target image using the ORB algorithm, detect key point positions using the FAST algorithm, and calculate description vectors of feature points using the BRIEF descriptor to obtain the two-dimensional coordinates of initial feature points; input the image into a shared backbone network of a single multi-task deep learning network to extract a global feature map F;

[0061] Based on the global feature map F, the following steps are performed by the multi-task branch network:

[0062] Combined with the initial feature point coordinates generated by the ORB algorithm, the two-dimensional coordinates of the final feature point are optimized and outputted. i ,y i )};

[0063] The edge feature map E is generated through the edge detection branch to assist in optimizing the positioning accuracy and robustness of the feature points;

[0064] The depth map D of the image is generated by the depth estimation branch, and the two-dimensional coordinates of the final feature points are mapped to the three-dimensional space to calculate the three-dimensional coordinates (X i , Y i , Z i ), the specific calculation formula is: Among them, (x i ,y i ) is the two-dimensional coordinate of the feature point, D(x i ,y i ) is the corresponding depth value, f x , f y is the focal length of the camera, c x , c y is the coordinate of the optical center of the image;

[0065] Multi-task loss function through joint optimization Weight balance training is performed on feature point detection, edge detection and depth estimation. The specific formula is: Among them, λ 1 ,λ 2 ,λ 3 are the weight coefficients of task loss respectively. In the electrical experiment teaching scenario, students observe the real circuit board through the system's augmented reality function. The system uses a single multi-task deep learning network that integrates the ORB algorithm to quickly identify the key positions of components such as resistors, capacitors, and wire interfaces on the circuit board, and accurately extract these feature points. At the same time, the system can also detect the edge contours of circuit board components to help distinguish the boundaries and connection relationships of different components.

[0066] In this embodiment, the steps of performing weighted fusion of the global feature map and the local feature points with a multi-head attention mechanism to optimize the expression and matching performance of the feature points are as follows:

[0067] The feature point information P is used as the query Q, and the feature vector of the global feature map F is used as the key K and value V. Perform attention calculation, where Q = W Q P, K = W K F、V=W V F.W Q , W K , W V is the learnable weight matrix, d k is the dimension of the key vector;

[0068] The attention mechanism is then expanded to a multi-head form, where multiple independent attention branches are calculated separately, and the results are concatenated and mapped back to the original dimension.

[0069] In this embodiment, the two-dimensional coordinates and three-dimensional coordinates of the feature points output by the multi-task learning module are combined with the camera internal parameters to solve the camera's posture information through the PnP algorithm, including the steps of rotating the matrix R and translating the vector T.

[0070] According to the camera's intrinsic parameter matrix K, the two-dimensional coordinates of the feature points are associated with the three-dimensional coordinates. The intrinsic parameter matrix K is defined as: Among them, f x and f y are the horizontal and vertical focal lengths of the camera, c x and c y is the optical center coordinate.

[0071] The two-dimensional coordinates (x i ,y i ) and the corresponding three-dimensional coordinates (X i , Y i , Z i) Input the PnP algorithm and solve the camera's pose information by minimizing the projection error. The expression is: Finally, the PnP algorithm is used to iteratively solve the camera's position information, including the rotation matrix R and the translation vector T.

[0072] In this embodiment, according to the rotation matrix R and the translation vector T, the coordinate system of the virtual three-dimensional model is transformed into the camera coordinate system, which is expressed as: cam =R·P world +T, where P world represents the point coordinates of the virtual 3D model in the world coordinate system, P cam Represents the point coordinates in the camera coordinate system after the pose transformation;

[0073] The virtual three-dimensional point P in the camera coordinate system is transformed into cam Projected onto the two-dimensional image plane, its expression is: img =K·P cam .

[0074] According to the two-dimensional coordinate p obtained by the projection img , rendering the virtual 3D model to the image plane position corresponding to the real scene, so that the virtual content is visually accurately aligned with the real object;

[0075] As the camera pose information changes in real time, the position and viewing angle of the virtual 3D model are dynamically updated to keep it synchronized with the real scene, achieving dynamic superposition and fusion display of virtual content and real scene. Finally, the final display result is output to complete the precise alignment and virtual-reality fusion display of the virtual 3D model and the real scene.

[0076] In electrical experiment teaching, circuit boards are used as teaching objects. Through augmented reality (AR) technology, virtual three-dimensional models (such as circuit component annotations, virtual wiring, dynamic current flow display, etc.) are superimposed on the real circuit board to help students understand the circuit connection principles and component functions.

[0077] When in use, the circuit board image is captured by the camera, and the feature point detection algorithm (such as fusion of ORB and deep learning network) is used to extract the feature points on the circuit board. Then, multiple feature points on the circuit board (such as resistors, capacitors, wire interfaces, etc.) are matched and associated with coordinates. Then, the point coordinates of the virtual three-dimensional model are converted from the world coordinate system to the camera coordinate system. Then, the three-dimensional points in the camera coordinate system are projected to the two-dimensional image plane through the camera intrinsic parameter matrix KKK. Then, the system accurately aligns the virtual resistor and current flow animation with the physical component position of the real circuit board and renders it on the image plane. When the user moves the camera or changes the viewing angle, the camera's posture information is updated in real time, and the position and viewing angle of the virtual three-dimensional model change synchronously to ensure the stability of the virtual-real fusion effect. For example, when the camera captures the resistor area on the circuit board, the system superimposes a virtual resistance value label (such as "R = 100Ω"). When the user observes the circuit wiring, the virtual current flow animation dynamically displays the flow direction of the current in the circuit, helping students to intuitively understand the current path.

[0078] In this embodiment, the steps of analyzing the user's learning behavior based on the interaction data and evaluating the learning effect are as follows:

[0079] Extract characteristic information related to learning behavior, such as operation frequency, duration, accuracy, and operation path, and analyze the user's learning behavior based on the extracted characteristic information, including the user's proficiency in operating virtual content, key focus areas, and number of incorrect operations;

[0080] Based on the interaction between the user and the virtual teaching content, the user's learning effect is evaluated to determine whether the user has mastered the relevant knowledge and operation points of the teaching target objects.

[0081] For example: In an electrical experiment scenario, the system monitors the number of times the user clicks on each component on the circuit board, the number of times the user sets the variable resistor, and the accuracy and time spent connecting the ammeter and voltmeter, and extracts the core feature information of the user's operation; for example, when the user frequently adjusts the parameters during the oscilloscope operation but fails to set them successfully, the system will mark this part as an "incorrect operation area". If the user repeatedly operates a component on the circuit board but the path is incorrect, the system determines that the user has not yet mastered the relevant knowledge points;

[0082] When the user completes the correct connection path in the ammeter operation task and the number of errors is lower than the set threshold, the system determines that the user has mastered the knowledge point; but in the resistor box setting task, if the number of errors is high and the operation takes a long time, the system determines that the user has not yet mastered the resistance adjustment method. When the system detects that the user's setting operation error rate on the resistor box reaches 70%, and the setting time is significantly higher than the average level, the system marks the area as "unmastered area" and prompts the user to practice further. The threshold setting is preset in the auxiliary system according to the actual situation.

[0083] This experiment focuses on the electrical experiment teaching scenario, and selects common electrical experiment equipment (such as circuit boards, oscilloscopes, ammeters, voltmeters, resistor boxes, etc.) as teaching targets, aiming to verify the performance advantages of this solution in feature point detection, edge detection, depth estimation, and pose estimation. Through comparative experiments, the accuracy, real-time performance, and robustness of this solution and existing technologies are evaluated.

[0084] 1) Dataset Content

[0085] For electrical experiment teaching, image data containing a variety of electrical experimental equipment is collected. The images cover different lighting conditions, viewing angles, occlusion levels and background complexities to ensure the diversity and challenge of the data set.

[0086] Teaching objectives:

[0087] Circuit board: contains the welding and wiring of different components, such as resistors, capacitors, and transistors. Oscilloscope: a screen and operation panel that displays different waveform signals. Voltmeter / ammeter: analog and digital display devices with different ranges. Resistor box: contains an adjustment device with multiple resistance levels. Ammeter and switch circuit combination: connection and switch control in complex circuits.

[0088] (2) Image acquisition details: Image resolution: 1920×10801920\times 10801920×1080. Number of images: 5000 in total, of which approximately 1000 were collected from each type of device.

[0089] Diverse conditions: Light variations: strong light, weak light, natural light.

[0090] Perspective changes: front, side, and oblique angle shooting. Occlusion: some device components are blocked by hands, cables, etc. Background complexity: simple background and experimental desktop background.

[0091] (3) The dataset is divided into training set: 70% (3,500 images) - used for network training. Validation set: 15% (750 images) - used for network parameter tuning and performance monitoring. Test set: 15% (750 images) - used for final evaluation of model performance.

[0092] Experimental equipment and training settings

[0093] (1) Hardware equipment

[0094] Image acquisition equipment: industrial-grade high-definition camera (1920×1080 resolution).

[0095] Computing equipment: CPU: Intel Core i9-12900K; GPU: NVIDIA RTX 4090 (24GB video memory); Memory: 32GB RAM

[0096] Software and Framework: Operating system: Ubuntu 20.04; Deep learning framework: PyTorch 2.0; Image processing library: OpenCV, NumPy

[0097] (2) Training parameter setting

[0098] Optimizer: Adam, Learning rate: Initial value is 0.001, using learning rate decay strategy (ReduceLROnPlateau) for dynamic adjustment. Batch size: 16, Number of training rounds: 50 rounds (Epochs), Loss function: Multi-task joint optimization loss function

[0099] Comparative Example 1: Single-task traditional feature extraction method: Use the ORB algorithm to extract feature points and output two-dimensional coordinates and description vectors. Directly input the feature points output by the ORB algorithm into the PnP algorithm to estimate the camera pose information.

[0100] Comparative Example 2: Multi-network independent task learning method: Independent deep learning networks are used to complete feature point detection, edge detection and depth estimation tasks, and each task is trained and optimized independently.

[0101]

[0102] The feature point detection accuracy of the traditional ORB method is 74.60%, which is limited by complex lighting and occlusion environments. The accuracy of the independent multi-network solution is improved to 84.20%, but the feature point detection is not robust enough without the ORB algorithm. The present invention improves the accuracy to 93.50% by integrating the ORB algorithm with the multi-task deep learning network, significantly optimizing the positioning and expression performance of the feature points.

[0103] Traditional methods do not have edge detection capabilities. The edge detection accuracy of the independent multi-network solution is 77.80%. The present invention achieves an edge detection accuracy of 89.60% through shared features and task branch optimization, further improving the positioning accuracy of feature points.

[0104] The depth estimation error of the independent multi-network solution is 3.12cm, which has a certain error. The present invention reduces the error to 1.87cm through task joint optimization and feature sharing, ensuring more accurate depth estimation.

[0105] The traditional ORB method has a large pose estimation error, with a rotation error of 2.8° and a translation error of 6.1cm. This invention combines high-precision feature points with three-dimensional coordinates and optimizes the PnP algorithm to reduce the rotation error to 0.9° and the translation error to 2.5cm.

[0106] The traditional ORB method has a running time of 38ms, but its performance is limited. The independent multi-network method has better performance, but the computational redundancy leads to a longer running time (102ms). The present invention achieves real-time performance while ensuring accuracy through shared backbone network and task joint optimization, with a running time of 63ms, which can meet the needs of teaching applications.

[0107] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. An intelligent teaching assistance system based on augmented reality, characterized in that: include: Image acquisition module: used to capture image data of teaching objects through a camera; Multi-task learning feature extraction module: Based on a single multi-task deep learning network, the ORB algorithm is integrated to extract local feature points of the teaching target image, and simultaneously complete edge detection and depth estimation tasks to generate two-dimensional coordinates, edge information and three-dimensional spatial coordinates of feature points; Multi-head attention feature fusion module: used to perform multi-head attention mechanism weighted fusion on global feature maps and local feature points to optimize the expression and matching performance of feature points; Camera pose estimation module: Combines the 2D and 3D coordinates of the feature points output by the multi-task learning module with the camera internal parameters, and uses the PnP algorithm to solve the camera pose information, including the rotation matrix. R and translation vectors T ; Virtual information overlay module: Based on the camera posture information, the 3D virtual model is accurately aligned and dynamically overlaid with the real scene to achieve virtual-real fusion display; Human-computer interaction module: used to support users to interact with virtual 3D models and store them through gesture recognition, voice control or touch operation; Learning behavior evaluation module: analyzes the user's learning behavior based on the interaction data and evaluates the learning effect.

2. The intelligent teaching auxiliary system based on augmented reality according to claim 1, characterized in that: The single multi-task deep learning network adopts the design of shared backbone network and task branch structure. The global feature map is extracted through the shared backbone network, and multiple task branches are set up on the basis of the backbone network. The specific structure includes Input layer, the input layer receives the image data of the teaching target object as the input of the network; Shared backbone network: The shared backbone network is used to extract multi-scale global feature maps as input for subsequent task branches; Multi-task branches: The multi-task branches are used to complete the three tasks of feature point detection, edge detection and depth estimation respectively; task joint optimization module: the joint optimization of task branches is realized through the multi-task loss function.

3. The intelligent teaching auxiliary system based on augmented reality according to claim 1, characterized in that: Based on a single multi-task deep learning network, the ORB algorithm is integrated to extract the local feature points of the teaching target image, and the edge detection and depth estimation tasks are completed at the same time. The steps to generate the two-dimensional coordinates, edge information and three-dimensional space coordinates of the feature points are as follows: Extract feature points from the input teaching target image using the ORB algorithm, detect key point positions using the FAST algorithm, and calculate description vectors of feature points using the BRIEF descriptor to obtain the two-dimensional coordinates of initial feature points; input the image into a shared backbone network of a single multi-task deep learning network to extract a global feature map F; Based on the global feature map F, the following steps are performed by the multi-task branch network: Combined with the initial feature point coordinates generated by the ORB algorithm, the two-dimensional coordinates of the final feature point are optimized and outputted. i ,y i )}; The edge feature map E is generated through the edge detection branch to assist in optimizing the positioning accuracy and robustness of the feature points; The depth map D of the image is generated by the depth estimation branch, and the two-dimensional coordinates of the final feature points are mapped to the three-dimensional space to calculate the three-dimensional coordinates (X i , Y i , Z i ), the specific calculation formula is: Among them, (x i ,y i ) is the two-dimensional coordinate of the feature point, D(x i ,y i ) is the corresponding depth value, f x , f y is the focal length of the camera, c x , c y is the coordinate of the optical center of the image; Multi-task loss function through joint optimization Weight balance training is performed on feature point detection, edge detection and depth estimation. The specific formula is: Among them, λ1, λ2, and λ3 are the weight coefficients of task loss respectively.

4. The intelligent teaching auxiliary system based on augmented reality according to claim 1, characterized in that: The steps of weighted fusion of the global feature map and local feature points using a multi-head attention mechanism to optimize the expression and matching performance of feature points are as follows: The feature point information P is used as the query Q, and the feature vector of the global feature map F is used as the key K and value V. Perform attention calculation, where Q = W Q P, K = W K F、V=W V F.W Q , W K , W V is the learnable weight matrix, d k is the dimension of the key vector; The attention mechanism is then expanded to a multi-head form, where multiple independent attention branches are calculated separately, and the results are concatenated and mapped back to the original dimension.

5. The intelligent teaching auxiliary system based on augmented reality according to claim 1, characterized in that: Combining the 2D coordinates and 3D coordinates of the feature points output by the multi-task learning module with the camera internal parameters, the PnP algorithm is used to solve the camera's pose information, including the steps of rotation matrix R and translation vector T. According to the camera's intrinsic parameter matrix K, the two-dimensional coordinates of the feature points are associated with the three-dimensional coordinates. The intrinsic parameter matrix K is defined as: Among them, f x and f y are the horizontal and vertical focal lengths of the camera, c x and c y is the optical center coordinate; The two-dimensional coordinates (x i ,y i ) and the corresponding three-dimensional coordinates (X i , Y i , Z i ) Input the PnP algorithm and solve the camera's pose information by minimizing the projection error. The expression is: Finally, the PnP algorithm is used to iteratively solve the camera's position information, including the rotation matrix R and the translation vector T.

6. The intelligent teaching auxiliary system based on augmented reality according to claim 1, characterized in that: According to the rotation matrix R and the translation vector T, the coordinate system of the virtual three-dimensional model is transformed into the camera coordinate system, and its expression is: cam =R·P world +T, where P world represents the point coordinates of the virtual 3D model in the world coordinate system, P cam Represents the point coordinates in the camera coordinate system after the pose transformation; The virtual three-dimensional point P in the camera coordinate system is transformed into cam Projected onto the two-dimensional image plane, its expression is: img =K·P cam ; According to the two-dimensional coordinate p obtained by the projection img , rendering the virtual 3D model to the image plane position corresponding to the real scene, so that the virtual content is visually accurately aligned with the real object; As the camera posture information changes in real time, the position and viewing angle of the virtual 3D model are dynamically updated to keep it synchronized with the real scene, realizing the dynamic superposition and fusion display of virtual content and real scene. Finally, the final display result is output to complete the precise alignment and virtual-reality fusion display of the virtual 3D model and the real scene.

7. The intelligent teaching auxiliary system based on augmented reality according to claim 1, characterized in that: The steps of analyzing the user's learning behavior based on the interaction data and evaluating the learning effect are as follows: Extracting feature information related to the learning behavior, and analyzing the user's learning behavior based on the extracted feature information; Based on the interaction between the user and the virtual teaching content, the user's learning effect is evaluated to determine whether the user has mastered the relevant knowledge and operation points of the teaching target objects.

Images