Novel course resource construction method and system based on augmented reality technology

Abstract

The invention relates to a novel course resource construction method and system based on an augmented reality technology, and belongs to the field of digital education resources. According to the method and the system, a mobile AR adaptive feature enhancement and efficient matching algorithm is adopted to obtain an identification image, and image feature detection and identification are carried out; estimating the pose of the camera, and determining the position and the pose of the virtual element in the actual space; modeling display content in course resources in advance, calling a corresponding model according to a feature matching result, and performing virtual fusion according to a camera pose estimation result; and establishing a mobile augmented reality interaction mode. According to the method, adaptive feature enhancement and feature point efficient matching are carried out through the improved AE-ORB algorithm, the matching effectiveness and the matching efficiency of the feature points are improved compared with those of other methods, and therefore it is guaranteed that students can enhance the understanding of learning content through the method, and the learning effect is improved.

Description

Technical Field

[0001] The present invention belongs to the field of digital educational resources and relates to a novel course resource construction method and system based on augmented reality technology. Background Art

[0002] The digital transformation of education has played a key, comprehensive and guiding role in promoting the high-quality development of my country's education. However, the limitations of existing digital educational resources and teaching methods in high-quality teaching practice in the context of digital transformation of education are becoming increasingly prominent. How to give full play to the technical advantages of digital resources such as new digital teaching materials and VR models and reduce the impact of changes in teaching methods requires rebuilding the education and teaching system business and giving students more control over their learning. However, the existing course resources lack sufficient resilience, making it difficult to provide students with opportunities to explore, create and interact with known or unknown things, and it is difficult to meet students' personalized and autonomous learning needs, which seriously restricts the cultivation of students' independent exploration and innovation capabilities. Therefore, it is of great significance to construct a new teaching resource system under digital transformation.

[0003] Augmented reality (AR) technology presents users with a realistic new world by superimposing computer-generated virtual information onto the real environment. With the rapid development of AR technology, new digital teaching resources based on AR technology have attracted widespread attention in the industry in recent years. Lv Xiaofeng proposed an intelligent classroom solution that integrates augmented reality (AR), image recognition, blockchain and big data analysis technology. The solution achieves centralized management of the system through a cloud platform, aiming to build an innovative digital teaching material development model and teaching method. Wang Jun proposed an intelligent teaching system for soil mechanics that integrates augmented reality (AR) technology to create an "immersive" learning experience. The system consists of three main modules: data storage module, data processing module and user interaction module. It not only promotes the interaction between students and teaching materials, but also supports communication between students and between students and teachers. Introducing AR technology into teaching materials can not only integrate multimedia resources such as sound, video and animation, but also display abstract text or flat images in three-dimensional and graphical ways to achieve real-time interaction.

[0004] Although traditional feature extraction and matching algorithms can realize basic AR functions, they are slow to extract features and their matching speed is not fast enough when dealing with complex scenes. This leads to poor user experience in real-time applications, especially on mobile devices.

[0005] In addition, conventional algorithms have low matching accuracy when dealing with image noise, lighting changes, and perspective changes. This can lead to inaccurate fusion of AR models and textbook images, affecting students' observation and analysis. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a new course resource construction method and system based on augmented reality technology.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A new course resource construction method based on augmented reality technology, the method comprising:

[0009] S1. Use mobile AR adaptive feature enhancement and efficient matching algorithm to obtain logo images, and perform image feature detection and recognition;

[0010] S2, perform camera pose estimation to determine the position and pose of the virtual element in the actual space;

[0011] S3. Model the display content in the course resources in advance, retrieve the corresponding model according to the feature matching results, and perform virtual fusion according to the camera pose estimation results;

[0012] S4. Establish mobile augmented reality interaction mode.

[0013] Further, in step S1, it includes the following steps:

[0014] S11, preprocessing the image, including denoising, blurring, filtering and image enhancement;

[0015] S12, inputting the preprocessed image into an image pyramid, and calculating the number of feature points to be extracted in each layer of the image pyramid;

[0016] S13, extracting feature points using the adaptive threshold FAST corner point extraction method, and using the BEBLID feature descriptor to describe the features to obtain a binary description vector;

[0017] S14. Obtain the final precise matching point through the designed BGR matching algorithm.

[0018] Further, in step S12, in each layer of the image pyramid, the number of feature points to be extracted for each grid is calculated by a grid division method, and the edge feature points of each layer of the image are screened to exclude the boundary area. The width and height dimensions of each layer of the pyramid image used for feature extraction are calculated as follows:

[0019] w inew =w i -2EdgeTh

[0020] h inew =h i -2EdgeTh

[0021] In the formula, w inewand h inew is the width and height of the image after removing the border, w i and h i is the width and height of the i-th layer image pyramid, EdgeTH is the size of the removed border;

[0022] Moreover, the grid division should be proportional to the image size, and the number of grids is positively correlated with the number of feature points required.

[0023] Further, in step S13, the threshold of the adaptive threshold FAST corner point extraction method is changed according to the overall pixel change of the image, wherein the adaptive change mode of the threshold is:

[0024] Get the pixel mean of the image, and then compare the grayscale of each point with the overall mean. The pixel comparison relationship of each image is the calculation formula of the new threshold:

[0025]

[0026] Where, μ is the scale factor, μ∈[0,1]; H, W are the image height and width; is the pixel mean of all points in the image;

[0027] When using the BEBLID feature descriptor for feature description, a square image area is taken with the feature point as the center, multiple image block pairs are randomly selected in the area, the average grayscale value of their pixels is calculated, and they are subtracted to obtain f(x). The weak learner can be constructed by the feature extraction function f and the threshold T:

[0028]

[0029] In the formula, h k (z) Depends on the feature extraction function f:X→R and the threshold T, selects the feature set through the enhancement algorithm, generates a strong feature description, converts the output of {+1,-1} into the output of {1,0}, and finally obtains the binary description vector of BEBLID.

[0030] Further, in step S14, the BGR matching algorithm first combines the brute force algorithm BF and the grid-based motion statistics method GMS to perform preliminary screening to distinguish correct matching points from incorrect matching points;

[0031] Then establish an evaluation function to measure the distance similarity between matching point pairs, and screen the matching point pairs according to the average value of the evaluation function of all matching point pairs. Among them, the matching point pairs with evaluation function values ​​greater than the average value are excluded, and the rest are retained;

[0032] The retained matching points are iteratively optimized using the RANSAC algorithm to determine the final accurate matching points.

[0033] Further, in step S2, the camera pose estimation includes the following process:

[0034] Let the world coordinates be (X W ,Y W ,Z W ), the camera coordinate system is (X C ,Y C ,Z C ), the image coordinate system is (x, y, z);

[0035] In the process of camera imaging, the conversion relationship between the world coordinate system and the camera coordinate system is first solved, that is, (X W ,Y W ,Z W ) and (X C ,Y C ,Z C ) The relationship between two points, this process involves multiple rotation and translation operations, the conversion formula is:

[0036]

[0037] in, Represents the transformation matrix from the world coordinate system to the camera coordinate system, R is the rotation matrix, t is the translation variable; L W Represents the transformation matrix from the world coordinate system to the camera coordinate system, which contains rotation and translation information;

[0038] When converting from the camera coordinate system to the image coordinate system, set the value of the camera coordinate system Z axis to the focal length F and calculate the Euclidean distance from the object to the image plane:

[0039]

[0040] Where, f represents the focal length;

[0041] The final step is to convert the physical coordinate system of the image into the pixel coordinate system. The two coordinate systems are on the same plane and are both two-dimensional coordinates. The conversion formula between the two is shown as follows:

[0042]

[0043] Among them, u and v are the horizontal and vertical coordinates in the image coordinate system, respectively, and d x ,d y They represent the ratio of the physical size to the pixel size in the x and y directions respectively, and u0 and v0 are the positions of the origin in the image coordinate system in the physical coordinate system.

[0044] The camera's external parameters determine the specific position of the virtual object rendered on the screen. First, the homography matrix is ​​calculated based on the pixel coordinates in the image and the four pairs of corresponding points in the world coordinates. Then, the optimized homography matrix is ​​used to deduce the camera's position and posture.

[0045] Furthermore, in step S3, 3ds Max is used to realistically restore the model in the textbook. The modeling of mechanical parts is combined with picture shooting and modeling tools, and the shape and effect are obtained through Mesh, Patch and NURBS modeling in 3DsMax; each part is rendered in layers, and the rendering process is smoothly optimized, and then material processing is performed to adjust material properties and glossiness, and then the model is established;

[0046] The corresponding model is retrieved according to the feature matching results, and virtual fusion is performed according to the camera pose estimation results. The system searches for a 3D model that matches the features in a pre-established model library based on the matching results; the 3D model is placed at the corresponding position in the real world according to the camera pose, and finally the virtual model is fused with the real-world scene using OpenGL rendering technology.

[0047] Further, in step S4, the screen touch is used to interact with the established virtual model, including rotating the virtual model up and down and left and right with a single finger, and zooming in and out with two fingers; the two-finger model scaling is based on obtaining the coordinates of the two fingers after initial contact and sliding on the screen, and calculating and comparing the change in the distance between the two touch points before and after. When the distance between the two touch points increases, the model is enlarged accordingly; otherwise, the model is reduced.

[0048] The present invention also proposes a novel course resource construction system based on augmented reality technology, which combines a cloud service platform to execute the aforementioned novel course resource construction method based on augmented reality technology, and comprises: a cloud platform and a terminal device;

[0049] The cloud platform includes Nginx load balancing module, security verification mechanism module, SpringBoot framework and Tomcat server. The SpringBoot framework has model library, video library and user library. The cloud platform also includes AR tracking registration module, image recognition algorithm module, model optimization algorithm module, model compression algorithm module and algorithm service module.

[0050] Among them, the Nginx load balancing module is used to distribute user requests, the security verification mechanism module is used to verify user identity, the SpringBoot framework and Tomcat server are used to provide Web services, the AR tracking and registration module is used to implement AR content tracking and registration, the image recognition algorithm module is used to identify image content, the model optimization algorithm module and the model compression algorithm module are used to optimize and compress 3D models, and the algorithm service module is used to provide various algorithm services;

[0051] Terminal devices include mobile phones, desktop computers and tablets, and terminal devices are connected to the cloud platform through 5G networks.

[0052] The beneficial effects of the present invention are:

[0053] The digital teaching resource construction technology based on AR of the present invention includes a digital resource AR system based on a cloud platform, a mobile AR three-dimensional tracking and registration technology, a digital teaching resource three-dimensional modeling, a mobile AR human-computer interaction technology, etc. At the algorithm level, the feature extraction speed, matching speed and matching accuracy are particularly optimized, thereby greatly enhancing the teaching effect. Students can scan the image model on the teaching material through a mobile terminal (mobile phone, tablet, etc.), and use the optimized algorithm to merge the cloud three-dimensional model with the teaching material two-dimensional model at an extremely fast matching speed. This fast matching process not only improves efficiency, but also significantly improves the matching accuracy due to the optimization of the algorithm in removing redundant and unreliable features. These technological advances not only enable students to observe and analyze three-dimensional models in depth from different angles, but also due to the high efficiency of the algorithm, students can obtain model information faster, observe and analyze the structure of the model and the details of each component more fluently, and provide students with a more intuitive, interactive and personalized learning environment.

[0054] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0056] Figure 1 It is an overall flow chart of the novel course resource construction method based on augmented reality technology of the present invention;

[0057] Figure 2 It is a schematic diagram of the feature point extraction process of the AE-ORB algorithm of the present invention;

[0058] Figure 3 are examples of feature point matching results from different algorithms, where: Figure 3 (a) is a matching example of SIFT algorithm. Figure 3 (b) is a matching example of the SURF algorithm, and 3(c) is a matching example of the traditional ORB algorithm. Figure 3 (d) is a matching example of the AE-ORB algorithm of the present invention;

[0059] Figure 4 It is a schematic diagram of the coordinate system conversion relationship of the present invention;

[0060] Figure 5 This is the overall framework diagram of the novel course resource construction system based on augmented reality technology of the present invention. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0062] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0063] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0064] See also Figure 1 to Figure 5 , which is a new course resource construction method and system based on augmented reality technology.

[0065] Example

[0066] This embodiment first provides a specific implementation process of a new course resource construction method based on augmented reality technology, which includes the following steps:

[0067] S1. Use mobile AR adaptive feature enhancement and efficient matching algorithm to obtain logo images, and perform image feature detection and recognition;

[0068] S2, perform camera pose estimation to determine the position and pose of the virtual element in the actual space;

[0069] S3. Model the display content in the course resources in advance, retrieve the corresponding model according to the feature matching results, and perform virtual fusion according to the camera pose estimation results;

[0070] S4. Establish mobile augmented reality interaction mode.

[0071] Currently, AR 3D registration and tracking includes two types of algorithms: based on markers and without markers. This embodiment adopts a registration and tracking algorithm based on markers, and its process principle is as follows: Figure 1 As shown, it mainly includes identification image acquisition, image feature detection and recognition, camera position estimation, virtual fusion and other links.

[0072] In step S1 of this embodiment, the traditional ORB algorithm often encounters problems such as insufficient texture information in the scene, insufficient brightness, and difficulty in ensuring that the extracted feature points are fully and evenly distributed in the image. In view of the above problems of the ORB algorithm, the present invention improves the ORB algorithm and proposes an adaptive feature enhancement and efficient matching AE-ORB algorithm (Adaptive and Boosted Efficient ORB), which strengthens key links such as image denoising, adaptive corner point extraction, feature description and feature matching. The AE-ORB algorithm specifically includes the following steps:

[0073] S11, preprocessing the image, including denoising, blurring, filtering and image enhancement; before feature extraction, the image must be preprocessed because uneven illumination may affect ORB feature extraction and image mapping. In this embodiment, a Gaussian filter is used to remove noise, and a 3×3 Gaussian kernel is applied to blur and filter the image, and then image enhancement is performed.

[0074] S12, input the preprocessed image into the image pyramid, calculate the number of feature points to be extracted in each layer of the image pyramid; in each layer of the image pyramid, calculate the number of feature points to be extracted for each grid by the grid division method. In order to ensure the accuracy of image matching, this embodiment screens the edge feature points of each layer of the image and excludes the boundary area to avoid matching failures caused by unstable edge feature points. At this time, the width and height formulas of each layer of the pyramid image that can actually be used for feature extraction are as follows:

[0075] w inew =w i -2EdgeTh

[0076] h inew =h i -2EdgeTh

[0077] In the formula, w inew and h inew is the width and height of the image after removing the border, w i and h i is the width and height of the i-th layer image pyramid, and EdgeTH is the size of the removed border.

[0078] The number of feature points in each layer is determined by the total number of layers of the image pyramid and the total number of feature points, so that feature points can be evenly and appropriately extracted from each layer of the pyramid. The grid division should be proportional to the image size, so that the distribution of feature points will be more uniform, and the number of grids is also positively correlated with the required number of feature points. In order to ensure that in each layer of the pyramid, the feature points can be evenly distributed in each grid and each layer can reach the predetermined number of feature points, the feature point extraction process is as follows: Figure 2 .

[0079] S13, the adaptive threshold FAST corner point extraction method is used to extract feature points, and the BEBLID (Boosted Efficient Binary Local Image Descriptor) feature descriptor is used for feature description to obtain a binary description vector; on the basis of gridding, the adaptive threshold FAST corner point extraction method is designed, and the threshold can be changed according to the overall pixel change of the image. First, the pixel mean of the image is obtained, and then the grayscale of each point is compared with the overall mean. The pixel comparison relationship of each image is the calculation formula of the new threshold:

[0080]

[0081] Where, μ is the scale factor, μ∈[0,1]; H, W are the image height and width; is the pixel mean of all points in the image.

[0082] On this basis, the BEBLID feature descriptor is used for feature description, replacing the original ORB's BRIEF feature matching algorithm. Take a square image area with the feature point as the center, randomly select multiple image block pairs in the area, calculate the average gray value of their pixels, and subtract them to get f(x). The weak learner can be constructed by the feature extraction function f and T:

[0083]

[0084] Where h k (z) depends on the feature extraction function f:X→R and the threshold T. By using the enhancement algorithm to selectively select the feature set and generate a strong feature description, the output of {+1, -1} is converted to the output of {1, 0}, and finally the binary description vector of BEBLID is obtained.

[0085] S14. Obtain the final precise matching points through the designed matching algorithm. A matching algorithm named BGR is used. The algorithm first performs preliminary screening by combining the brute force algorithm BF (Brute Force) and the grid-based motion statistics method GMS (Grid-based Motion Statistics) to distinguish correct matching points from incorrect matching points. Based on the matching results obtained through screening, an evaluation function is established to measure the distance similarity between matching point pairs. If a value exceeds the average value of the evaluation function of all paired point pairs, the corresponding target pairing will be excluded. After screening, the remaining matching points will be iteratively optimized through the RANSAC algorithm to determine the final precise matching points.

[0086] This embodiment conducts a comparative experiment on the AE-ORB algorithm in step S1, wherein this experiment uses the TUM public dataset to test the AE-ORB algorithm. The operating system used by the experimental platform is Windows 11, the CPU is AMD Ryzen 75800H, the frequency is 3.20GHz, and the graphics card is GTX1650. In this experiment, this embodiment selects desktop object pictures with rich textures as the dataset, while the texture of the floor is relatively simple, which is suitable for the research needs of this embodiment. Through the selection of this dataset, this embodiment can better evaluate the performance of the algorithm under different texture conditions.

[0087] Table 1 shows the feature point extraction and matching results of different algorithms and the algorithm of the present invention under scenario 1:

[0088] Table 1

[0089]

[0090] As shown in Table 1, the matching rate of the AE-ORB algorithm for scene 1 is 73.75%, which is nearly 30% higher than that of the ORB algorithm. Although the number of feature points and matching points has decreased, the matching rate is much higher than that of other algorithms, indicating that the algorithm has done a good job in optimizing the removal of redundant and unreliable features. The matching time is only 0.0023 seconds, which is much lower than that of other algorithms. The matching process is efficient, which proves the effectiveness of the algorithm in some application scenarios.

[0091] Figure 3 Figure 2 shows examples of feature point matching results from different algorithms, where: Figure 3 (a) is a matching example of SIFT algorithm. Figure 3 (b) is a matching example of the SURF algorithm, and 3(c) is a matching example of the traditional ORB algorithm. Figure 3 (d) is a matching example of the AE-ORB algorithm of the present invention. Figure 3 (c) is the result of the traditional ORB matching algorithm. Figure 3 The two images in (c) are matching image pairs, and the connected line segments are matching feature point pairs of the matching images. Figure 3 In (c), the positions of the two end points of the left and right green line segments are inconsistent. The traditional ORB method mainly focuses on the area with rich texture when extracting feature points. This feature enables it to match more feature points. However, it also leads to too dense matching points and the existence of mismatches. In contrast, the AE-ORB algorithm of the present invention is Figure 3 As shown in (d), the accuracy of feature matching is successfully improved by reducing the number of feature points while maintaining real-time performance.

[0092] Table 2 shows the matching rates of different algorithms in different scenarios:

[0093] Table 2

[0094]

[0095] In the data display of Table 2, this embodiment compares the matching performance of the traditional ORB algorithm, SURF algorithm, SIFT algorithm and AE-ORB algorithm in various scenes. In the test of scene 1, the matching accuracy of the AE-ORB algorithm reached 73.75%, while the accuracy of the ORB algorithm was 34.55%, the SURF algorithm was 28.42%, and the SIFT algorithm was 30.46%; when observing scene 2, although the matching efficiency of the AE-ORB algorithm decreased, it still led with a rate of 58.62%. At the same time, the matching rate of the ORB algorithm was only 33.34%. Through these comparisons, it can be seen that the matching rate of the AE-ORB algorithm is better than that of other algorithms, and a higher matching rate can be obtained.

[0096] Table 3 shows the matching time of different algorithms in different scenarios:

[0097] Table 3

[0098]

[0099] In Table 3, this embodiment compares the time required for matching of the traditional ORB algorithm, SURF algorithm, SIFT algorithm and AE-ORB algorithm when processing different images. It can be seen that the matching time of the improved ORB algorithm in different scenes is significantly reduced, which is much faster than the original ORB, SURF and SIFT. This shows that the algorithm has made significant improvements in performance optimization, especially in matching efficiency.

[0100] In step S2 of this embodiment, in order to achieve real-time integration of virtual objects and the real world in an augmented reality environment, the key step is to accurately and quickly determine the position and posture of the virtual elements in the actual space. This process is called pose estimation. It includes the following processes:

[0101] Let the world coordinates be (X W ,Y W ,Z W ), the camera coordinate system is (X C ,Y C ,Z C ), the image coordinate system is (x, y, z), and the conversion relationship between the three is as follows Figure 4 In the process of camera imaging, we first need to solve the conversion relationship between the world coordinate system and the camera coordinate system, that is, (X W ,Y W ,Z W ) and (X C ,Y C ,Z C ) The relationship between two points, this process involves multiple rotation and translation operations, the conversion formula is:

[0102]

[0103] in, Represents the transformation matrix from the world coordinate system to the camera coordinate system, R is the rotation matrix, t is the translation variable; L W Represents the transformation matrix from the world coordinate system to the camera coordinate system, which contains rotation and translation information;

[0104] When converting from the camera coordinate system to the image coordinate system, in order to fix the value of the Z axis of the camera coordinate system, its value is set to the focal length F. The following formula can be used to obtain the Euclidean distance from the object to the image plane.

[0105]

[0106] Where, f represents the focal length;

[0107] The final step in completing the camera imaging process is to convert the physical coordinate system of the image into the pixel coordinate system. These two coordinate systems are on the same plane and are both two-dimensional coordinates. The conversion formula between the two is shown below:

[0108]

[0109] Among them, u and v are the horizontal and vertical coordinates in the image coordinate system, respectively, and d x ,d y They represent the ratio of the physical size to the pixel size in the x and y directions respectively, and u0 and v0 are the positions of the origin in the image coordinate system in the physical coordinate system.

[0110] The camera external parameters determine where the virtual object is rendered on the screen. To quickly and accurately determine the camera pose, this embodiment first calculates the homography matrix based on the pixel coordinates in the image and the four pairs of corresponding points in the world coordinates. Then, the optimized homography matrix is ​​used to derive the camera pose.

[0111] In step S3 of the present embodiment, the realism of the model directly affects the sensory experience of the user. This system is mainly used in electronic textbooks for process control engineering colleges and universities. Due to the complex structure of the equipment model, in order to ensure that the system display effect is more realistic, this embodiment uses 3ds Max to perform a 1:1 real restoration of the model. The modeling of mechanical parts is mainly combined with picture shooting and modeling tools. Taking the regulating valve as an example, the color and texture of the regulating valve are taken from the pictures, while the shape and effect are obtained through Mesh, Patch and NURBS modeling in 3DsMax. Due to the different line thicknesses of each component, layered rendering is required. The model is relatively fine, resulting in a long rendering time, so it needs to be optimized. First, after selecting the model, use the "Turbo Smooth" option in the "Modify" menu for smooth optimization, then use the "Material Editor" to add materials to the model, and finally adjust the material properties and glossiness to achieve the desired effect.

[0112] In order to avoid too many GameObjects and difficulty in finding them during program development, related parts without animations also need to be grouped. The grouping method should be divided according to the model structure. After the grouping is completed, after the model is exported in fbx format, in order to facilitate processing and editing in Unity, a new "Model" folder can be created under the "Assets" folder of the Unity project to store the model resources, and the model can be imported.

[0113] Then, the corresponding model is retrieved according to the feature matching results, and virtual fusion is performed according to the camera pose estimation results.

[0114] Once the feature matching is completed, the system will search for 3D models that match these features in the model library based on the matching results. After the feature matching, a set of feature points and their descriptors will be obtained. The application uses the descriptor as a query condition and sends an SQL statement to the MySQL database to find matching feature records. A special data table is designed in the database to store feature information, including feature descriptors and the model ID associated with it. The database returns the matching feature record, and the AR application retrieves the corresponding 3D model information in another data table based on the model ID in the record, including the model's storage location, metadata, etc.

[0115] At the same time, through feature matching, the camera's position and posture can be estimated. Then, based on the camera's position and posture, the three-dimensional model can be correctly placed at the corresponding position in the real world. After the model is correctly placed, OpenGL rendering technology is used to ensure that the virtual model is seamlessly integrated with the real-world scene.

[0116] In step S4 of this embodiment, the system uses a mobile augmented reality interaction method to provide students with a scene experience that integrates the real and the virtual. Users can use screen touch to achieve a variety of interactive functions for the regulating valve model: single-finger up and down, left and right rotation, and double-finger zooming in and out. Users can easily rotate and zoom models such as regulating valves through gesture control, thereby gaining a deeper understanding of their details and features and enhancing the interactive experience. Table 4 shows a schematic diagram of touch screen interaction gestures:

[0117] Table 4

[0118]

[0119] In Table 4, the principle of two-finger model scaling is based on obtaining the coordinates of the initial contact and sliding of two fingers on the screen, and calculating the change in the distance between the two touch points before and after the comparison. When the distance between the two touch points increases, the model is enlarged accordingly; otherwise, the model is reduced. Specifically, when the user pinches or expands the screen with two fingers, the system monitors the movement trajectory of each finger in real time, calculates the change in the distance between the fingers, and adjusts the size of the model accordingly. Once the video QR code is recognized, click the play button, and the system will obtain the relevant video data stored on the server and play it.

[0120] Users can control the speed and volume of video playback. Students can watch related video materials while reading textbooks, thereby enhancing learning effects and depth of understanding. They can also pause for more detailed understanding and note-taking to meet personalized learning needs.

[0121] This embodiment also proposes a new course resource construction system based on augmented reality technology based on the aforementioned new course resource construction method based on augmented reality technology, which executes the aforementioned new course resource construction method based on augmented reality technology based on a cloud platform. The display of augmented reality effects involves a large number of complex operations, such as textbook image collection, three-dimensional tracking registration, model rendering and drawing, etc. In order to optimize learners' demand for loading a large number of virtual digital learning resources during use, a digital resource AR system architecture based on a cloud platform, such as Figure 5 As shown in the figure, complex calculations and services are implemented in the cloud server, such as AR registration and tracking, image recognition algorithms, model optimization and compression algorithms, etc., and various resources are integrated with the help of the cloud platform.

[0122] In order to cope with the multi-user simultaneous access requirements of the cloud service platform, this system uses Nginx for load balancing. Nginx is a powerful HTTP / reverse proxy network server with load balancing function. In terms of performance, Nginx supports more concurrent connections and provides efficient connection speed; in terms of installation and configuration, Nginx is simple and flexible, supports hot deployment, and does not require restarting the service when upgrading the software version and changing the configuration.

[0123] The Nginx Web server can serve as the middle layer to process requests from the cloud platform and interact with the back-end Tomcat service. The included algorithm modules (including AR tracking registration, image recognition algorithm, model optimization algorithm, model compression algorithm, and algorithm service) constitute the core functions of the system and are responsible for processing AR-related technologies and data.

[0124] Use the SpringBoot framework to develop a RESTful API and provide an interactive interface with the client. Tomcat server is used as a Servlet container to run the SpringBoot application, process requests from the NginxWeb server, and interact with the database. The database contains the following modules: Model library: stores 3D model-related data. Video library: stores video resources related to teaching content. User library: stores user information, processes user requests, and provides personalized teaching resources and services. Use MySQL relational database to ensure data consistency and integrity.

[0125] Static files: stored in the Web service layer, used to provide static resources, such as HTML, CSS, JavaScript files, etc.

[0126] As a communication network, the 5G network connects the cloud platform and various client devices (such as mobile phones, tablets, and pads). It ensures fast data transmission and real-time interaction. In addition, in order to eliminate or avoid the risk of illegal access and illegal theft and illegal dissemination of model data during the transmission process of uploading model data on the cloud service platform, and to ensure that users can use the platform's services with confidence, user key management and data security service mechanisms are adopted to effectively protect data security.

[0127] Specifically, the new course resource construction system based on augmented reality technology includes a cloud platform and terminal devices. The cloud platform includes an Nginx load balancing module, a security verification mechanism module, a SpringBoot framework, and a Tomcat server. The SpringBoot framework is equipped with a model library, a video library, and a user library. The cloud platform also includes an AR tracking registration module, an image recognition algorithm module, a model optimization algorithm module, a model compression algorithm module, and an algorithm service module. The terminal devices include mobile phones, desktop computers, and tablets. The terminal devices are connected to the cloud platform through a 5G network. The Nginx load balancing module is used to distribute user requests, the security verification mechanism module is used to verify user identity, the SpringBoot framework and the Tomcat server are used to provide Web services, the AR tracking registration module is used to track and register AR content, the image recognition algorithm module is used to identify image content, the model optimization algorithm module and the model compression algorithm module are used to optimize and compress 3D models, and the algorithm service module is used to provide various algorithm services.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A new course resource construction method based on augmented reality technology, characterized by: The method comprises: S1. Use mobile AR adaptive feature enhancement and efficient matching algorithm to obtain logo images, and perform image feature detection and recognition; S2, perform camera pose estimation to determine the position and pose of the virtual element in the actual space; S3. Model the display content in the course resources in advance, retrieve the corresponding model according to the feature matching results, and perform virtual fusion according to the camera pose estimation results; S4. Establish mobile augmented reality interaction mode.

2. According to claim 1, a new type of course resource construction method based on augmented reality technology is characterized by: In step S1, it includes the following steps: S11, preprocessing the image, including denoising, blurring, filtering and image enhancement; S12, inputting the preprocessed image into an image pyramid, and calculating the number of feature points to be extracted in each layer of the image pyramid; S13, extracting feature points using the adaptive threshold FAST corner point extraction method, and using the BEBLID feature descriptor to describe the features to obtain a binary description vector; S14. Obtain the final precise matching point through the designed BGR matching algorithm.

3. The novel course resource construction method based on augmented reality technology according to claim 2 is characterized by: In step S12, in each layer of the image pyramid, the number of feature points to be extracted in each grid is calculated by a grid division method, and the edge feature points of each layer of the image are screened to exclude the boundary area. The width and height of each layer of the pyramid image used for feature extraction are calculated as follows: In inew =in i -2EdgeTh h inew =h i -2EdgeTh In the formula, w inew and h inew is the width and height of the image after removing the border, w i and h i is the width and height of the i-th layer image pyramid, and EdgeTH is the size of the removed border; Moreover, the grid division should be proportional to the image size, and the number of grids is positively correlated with the number of feature points required.

4. The novel course resource construction method based on augmented reality technology according to claim 3 is characterized by: In step S13, the threshold of the adaptive threshold FAST corner point extraction method is changed according to the overall pixel change of the image, wherein the adaptive change mode of the threshold is: Get the pixel mean of the image, and then compare the grayscale of each point with the overall mean. The pixel comparison relationship of each image is the calculation formula of the new threshold: Where, μ is the scale factor, μ∈[0,1]; H, W are the image height and width; I is the pixel mean of all points in the image; When using the BEBLID feature descriptor for feature description, a square image area is taken with the feature point as the center, multiple image block pairs are randomly selected in the area, the average grayscale value of their pixels is calculated, and they are subtracted to obtain f(x). The weak learner can be constructed by the feature extraction function f and the threshold T: In the formula, h k (z) Depends on the feature extraction function f:X→R and the threshold T, selects the feature set through the enhancement algorithm, generates a strong feature description, converts the output of {+1,-1} into the output of {1,0}, and finally obtains the binary description vector of BEBLID.

5. The novel course resource construction method based on augmented reality technology according to claim 4 is characterized by: In step S14, the BGR matching algorithm first combines the brute force algorithm BF and the grid-based motion statistics method GMS to perform preliminary screening to distinguish correct matching points from incorrect matching points; Then establish an evaluation function to measure the distance similarity between matching point pairs, and screen the matching point pairs according to the average value of the evaluation function of all matching point pairs. Among them, the matching point pairs with evaluation function values ​​greater than the average value are excluded, and the rest are retained; The retained matching points are iteratively optimized using the RANSAC algorithm to determine the final accurate matching points.

6. The novel course resource construction method based on augmented reality technology according to claim 2 is characterized by: In step S2, the camera pose estimation includes the following process: Let the world coordinates be (X W ,Y W ,Z W ), the camera coordinate system is (X C ,Y C ,Z C ), the image coordinate system is (x, y, z); In the process of camera imaging, the conversion relationship between the world coordinate system and the camera coordinate system is first solved, that is, (X W ,Y W ,Z W ) and (X C ,Y C ,Z C ) The relationship between two points, this process involves multiple rotation and translation operations, the conversion formula is: in, Represents the transformation matrix from the world coordinate system to the camera coordinate system, R is the rotation matrix, t is the translation variable; L W Represents the transformation matrix from the world coordinate system to the camera coordinate system, which contains rotation and translation information; When converting from the camera coordinate system to the image coordinate system, set the value of the camera coordinate system Z axis to the focal length F and calculate the Euclidean distance from the object to the image plane: Where, f represents the focal length; The final step is to convert the physical coordinate system of the image into the pixel coordinate system. The two coordinate systems are on the same plane and are both two-dimensional coordinates. The conversion formula between the two is shown as follows: Among them, u and v are the pixel coordinates in the image coordinate system, and d x ,d y are the physical sizes of each pixel in the image coordinate system in the x and y directions, respectively, and u0 and v0 are the positions of the origin of the image coordinate system in the pixel coordinate system; The camera's external parameters determine the specific position of the virtual object rendered on the screen. First, the homography matrix is ​​calculated based on the pixel coordinates in the image and the four pairs of corresponding points in the world coordinates. Then, the optimized homography matrix is ​​used to deduce the camera's position and posture.

7. The novel course resource construction method based on augmented reality technology according to claim 6 is characterized by: In step S3, 3ds Max is used in advance to realistically restore the model in the textbook. The modeling of mechanical parts is combined with picture shooting and modeling tools, and the shape and effect are obtained through Mesh, Patch and NURBS modeling in 3DsMax; each part is rendered in layers, and the rendering process is smoothly optimized, and then material processing is performed to adjust material properties and glossiness, and then the model is established; The corresponding model is retrieved according to the feature matching result, and virtual fusion is performed according to the camera pose estimation result. The system searches for a 3D model matching the feature in a pre-established model library according to the matching result. According to the camera's position, the 3D model is placed at the corresponding position in the real world, and finally the virtual model is integrated with the real-world scene using OpenGL rendering technology.

8. The novel course resource construction method based on augmented reality technology according to claim 7 is characterized by: In step S4, the screen touch is used to interact with the established virtual model, including rotating the virtual model up and down and left and right with a single finger, and zooming in and out with two fingers; the two-finger model zooming is based on obtaining the coordinates of the two fingers after the initial contact and sliding on the screen, and calculating and comparing the change in the distance between the two touch points before and after. When the distance between the two touch points increases, the model is enlarged accordingly; otherwise, the model is reduced.

9. A new course resource construction system based on augmented reality technology, characterized by: The system is combined with a cloud service platform to execute the novel course resource construction method based on augmented reality technology as described in any one of claims 1 to 8, and comprises: a cloud platform and a terminal device; The cloud platform includes Nginx load balancing module, security verification mechanism module, SpringBoot framework and Tomcat server. The SpringBoot framework has model library, video library and user library. The cloud platform also includes AR tracking registration module, image recognition algorithm module, model optimization algorithm module, model compression algorithm module and algorithm service module. Among them, the Nginx load balancing module is used to distribute user requests, the security verification mechanism module is used to verify user identity, the SpringBoot framework and Tomcat server are used to provide Web services, the AR tracking and registration module is used to implement AR content tracking and registration, the image recognition algorithm module is used to identify image content, the model optimization algorithm module and the model compression algorithm module are used to optimize and compress 3D models, and the algorithm service module is used to provide various algorithm services; Terminal devices include mobile phones, desktop computers and tablets, and terminal devices are connected to the cloud platform through 5G networks.

Images