AR-based cultural relic restoration and real-time interaction system
By combining AR glasses with modules for cultural relic revitalization and restoration, object recognition, and intelligent character interaction, the problems of delayed cultural relic recognition and inaccurate recommendations in complex environments have been solved. This has enabled accurate restoration and precise recommendations of cultural relics, thus improving the user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 北京中嘉空间展示设计有限公司
- Filing Date
- 2025-02-06
- Publication Date
- 2026-05-12
AI Technical Summary
Existing AR glasses suffer from delays or errors in object recognition in complex environments or when dealing with fast-moving objects. In particular, the processing speed decreases and resources are unevenly distributed in multi-object recognition scenarios, affecting the restoration effect of cultural relics and the accuracy of user recommendations. In the initial stage, the quality of cultural relic data is not high.
The system uses AR glasses combined with modules for cultural relic revitalization and restoration, object recognition, and intelligent character interaction. It acquires images of cultural relics through cameras for missing parts repair and 3D modeling, uses object recognition models to determine the characteristic data of cultural relics, and stores and displays cultural relic information through a database. The intelligent character interaction module provides real-time recommendations for voice introductions and interactive content.
It enables accurate identification, restoration, and precise recommendation of cultural relics, enhancing the user's immersion and interactive experience, and ensuring the accuracy of cultural relic restoration and the precision of recommendations.
Smart Images
Figure CN122018670A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of cultural relic restoration technology, and in particular to an AR-based cultural relic restoration and real-time interactive system. Background Technology
[0002] AR glasses require real-time perception and analysis of the real world. Current technology suffers from delays or misidentifications in complex environments or when dealing with rapidly moving objects. In multi-object recognition scenarios, AR glasses need to process information from multiple objects simultaneously, which can lead to decreased processing speed or uneven resource allocation. This is especially true when multiple objects interfere with each other, affecting recognition accuracy and resulting in poor restoration of cultural relics. In the initial stages or in specific fields, cultural relic data is very limited and often affected by environmental, equipment, and other factors, leading to low-quality or insufficient data collected on-site, resulting in poor recommendations for users. Summary of the Invention
[0003] The purpose of this application is to provide an AR-based system for the restoration and real-time interaction of cultural relics, in order to solve the problem of the inability to accurately identify, restore, and precisely recommend cultural relics.
[0004] To achieve the above objectives, this application provides the following solution:
[0005] This application provides an AR-based cultural relic restoration and real-time interactive system, including: AR glasses and a cultural relic activation and restoration module, a physical object recognition module, an intelligent character interaction module, and a database deployed in the AR glasses; the cultural relic activation and restoration module and the physical object recognition module are respectively connected to the database;
[0006] The cultural relic activation and restoration module is used for:
[0007] Acquire images of cultural relics captured by the camera of AR glasses;
[0008] Based on the image of the cultural relic, missing parts are repaired and three-dimensional modeling is performed to obtain a three-dimensional model of the cultural relic.
[0009] The three-dimensional model of the cultural relic is displayed to the user through the AR glasses;
[0010] The database is used to store the three-dimensional models of each of the cultural relics and the corresponding cultural relic information;
[0011] The object recognition module is used for:
[0012] Acquire images of cultural relics captured by the camera of AR glasses;
[0013] Using the built-in entity recognition model, the feature data of the cultural relic are determined based on the image of the cultural relic;
[0014] Based on the feature data, the corresponding 3D model of the cultural relic and the cultural relic information stored in the database are determined.
[0015] The three-dimensional model of the cultural relic and its information are displayed to the user through the AR glasses.
[0016] The intelligent character interaction module is used for:
[0017] Follow the user's route and obtain user data in real time;
[0018] Based on the user data, the system will recommend audio introductions and interactive content about cultural relics to users in real time.
[0019] Optionally, the AR-based cultural relic restoration and real-time interactive system further includes: a central control unit; the central control unit is connected to the intelligent character interaction module;
[0020] The central control unit is used to collect the user data.
[0021] Optionally, based on the image of the cultural relic, missing parts are repaired and three-dimensional modeling is performed to obtain a three-dimensional model of the cultural relic, including:
[0022] The image of the cultural relic is segmented to obtain a segmented image of the cultural relic.
[0023] The segmented cultural relic image is denoised to obtain the denoised cultural relic image;
[0024] Using the built-in modeling software, the missing parts of the cultural relics are repaired and three-dimensionally modeled based on the denoised images of the cultural relics, resulting in a cultural relic restoration model.
[0025] Determine the texture map based on the image of the cultural relic;
[0026] The three-dimensional model of the cultural relic is determined based on the cultural relic restoration model and the texture map.
[0027] Optionally, the modeling software includes 3ds Max and Maya.
[0028] Optionally, using a built-in entity recognition model, based on the image of the cultural relic, the characteristic data of the cultural relic are determined, including:
[0029] The image of the cultural relic is cleaned and denoised to obtain the processed image of the cultural relic;
[0030] The processed images of cultural relics are input into the entity recognition model to obtain the feature data of the cultural relics.
[0031] Optionally, the process of determining the entity recognition model includes:
[0032] Obtain a training set; the training set includes: multiple sample artifact images and corresponding artifact feature data;
[0033] Constructing a convolutional neural network;
[0034] The convolutional neural network is trained using the training set to obtain the entity recognition model.
[0035] Optionally, the convolutional neural network is trained using the training set to obtain the entity recognition model, including:
[0036] The convolutional neural network is trained using images of each sample cultural relic as input and the corresponding feature data of the cultural relic as output to obtain the entity recognition model.
[0037] Optionally, the entity recognition model can be optimized and updated in real time using the expanded training set.
[0038] Optionally, based on the user data, the system recommends voice introductions and interactive content about cultural relics to users in real time, including:
[0039] Using a recommendation algorithm, the system recommends audio descriptions and interactive content about cultural relics to users in real time based on the user data.
[0040] Optionally, the recommendation algorithm includes: content-based recommendation, collaborative filtering recommendation, and hybrid recommendation.
[0041] According to the specific embodiments provided in this application, the following technical effects are disclosed:
[0042] This application discloses an AR-based system for cultural relic restoration and real-time interaction, comprising: AR glasses and a cultural relic activation and restoration module, an object recognition module, an intelligent character interaction module, and a database deployed within the AR glasses; the cultural relic activation and restoration module and the object recognition module are respectively connected to the database; the cultural relic activation and restoration module is used to: acquire images of cultural relics captured by the camera of the AR glasses; perform missing repair and 3D modeling based on the cultural relic images to obtain a 3D model of the cultural relic; and display the 3D model of the cultural relic to the user through the AR glasses; the database is used to store the 3D models of each cultural relic and the corresponding cultural relic information; the object recognition module is used to: acquire images of cultural relics captured by the camera of the AR glasses; use a built-in entity recognition model to determine the feature data of the cultural relic based on the cultural relic images; based on the feature data, determine the corresponding 3D model of the cultural relic and the cultural relic information stored in the database; and display the 3D model of the cultural relic and the cultural relic information to the user through the AR glasses; the intelligent character interaction module is used to: follow the user's route and acquire user data in real time; and recommend voice introductions and interactive content of cultural relics to the user in real time based on the user data. This application utilizes the cultural relic activation and restoration module, the object recognition module, and the intelligent character interaction module to achieve accurate identification, restoration, and precise recommendation of cultural relics. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 A schematic diagram of the structure of an AR-based cultural relic restoration and real-time interactive system provided in one embodiment of this application. Detailed Implementation
[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0046] The purpose of this application is to provide an AR-based system for the restoration and real-time interaction of cultural relics, which aims to achieve accurate identification, restoration, and precise recommendation of cultural relics.
[0047] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0048] In one exemplary embodiment, such as Figure 1 As shown, an AR-based system for cultural relic restoration and real-time interaction is provided, including: AR glasses and a cultural relic activation and restoration module, an object recognition module, an intelligent character interaction module, and a database deployed in the AR glasses; the cultural relic activation and restoration module and the object recognition module are respectively connected to the database.
[0049] The cultural relic revitalization and restoration module is used for:
[0050] Acquire images of cultural relics captured by the camera of AR glasses.
[0051] Based on the images of cultural relics, missing parts are repaired and 3D modeling is performed to obtain a 3D model of the cultural relics.
[0052] The three-dimensional model of the cultural relic is displayed to the user through AR glasses.
[0053] The database is used to store the three-dimensional models of each cultural relic and the corresponding cultural relic information.
[0054] The object recognition module is used for:
[0055] Acquire images of cultural relics captured by the camera of AR glasses.
[0056] Using a built-in entity recognition model, the feature data of cultural relics are determined based on their images.
[0057] Based on feature data, the corresponding 3D model of the cultural relic and its information stored in the database are determined.
[0058] The three-dimensional model of the cultural relic and its information are displayed to the user through AR glasses.
[0059] The intelligent character interaction module is used for:
[0060] Follow the user's route and obtain user data in real time.
[0061] Based on user data, we recommend audio introductions and interactive content about cultural relics to users in real time.
[0062] As an optional implementation, the AR-based cultural relic restoration and real-time interactive system also includes: a central control unit; and a central control unit connected to an intelligent character interaction module.
[0063] The central control unit is used to collect user data.
[0064] As an optional implementation method, missing parts are repaired and 3D modeling is performed based on the image of the cultural relic to obtain a 3D model of the cultural relic, including:
[0065] Image contour segmentation is performed on the cultural relic image to obtain the segmented cultural relic image.
[0066] The segmented image of the cultural relic is denoised to obtain the denoised image of the cultural relic.
[0067] Using the built-in modeling software, the missing parts of the cultural relics are repaired and three-dimensionally modeled based on the denoised images of the relics, resulting in a cultural relic restoration model.
[0068] Texture mapping is determined based on images of cultural relics.
[0069] Based on the artifact restoration model and texture mapping, the three-dimensional model of the artifact is determined.
[0070] As an alternative implementation, modeling software includes 3ds Max and Maya.
[0071] As an optional implementation, a built-in entity recognition model is used to determine the feature data of the cultural relic based on the image, including:
[0072] Data cleaning and noise reduction are performed on the images of cultural relics to obtain the processed images.
[0073] The processed images of cultural relics are input into the entity recognition model to obtain the feature data of the cultural relics.
[0074] As an optional implementation method, the process of determining the entity recognition model includes:
[0075] Obtain the training set; the training set includes: multiple sample images of cultural relics and the corresponding feature data of the cultural relics.
[0076] Construct a convolutional neural network.
[0077] The convolutional neural network is trained using the training set to obtain an entity recognition model.
[0078] As an optional implementation, a convolutional neural network is trained using a training set to obtain an entity recognition model, including:
[0079] Using images of each cultural relic as input and the corresponding feature data of the cultural relic as output, a convolutional neural network is trained to obtain an entity recognition model.
[0080] As an optional implementation, the entity recognition model is optimized and updated in real time using the expanded training set.
[0081] As an optional implementation, based on user data, real-time recommendations of audio descriptions and interactive content about cultural relics are provided to users, including:
[0082] Using recommendation algorithms, the system provides users with real-time audio introductions and interactive content about cultural relics based on user data.
[0083] As an alternative implementation method, recommendation algorithms include: content-based recommendation, collaborative filtering recommendation, and hybrid recommendation.
[0084] Specifically, the detailed introduction and implementation steps of each module are as follows.
[0085] 1. Cultural Relics Activation and Restoration Module: Utilizing image recognition and tracking technology, this module identifies real-world cultural relics, performs missing part repair and 3D modeling based on the relics' images, and presents the 3D model to the user virtually. Gesture recognition technology allows users to interact with the virtual relics, providing historical background information and enhancing immersion and interactive experience.
[0086] Based on the collected images of cultural relics, computer graphics software and tools, such as 3ds Max and Maya, are used to create 3D models of the missing or damaged parts of the relics, resulting in restoration models. During the modeling process, historical data about the relics, characteristics of similar relics, and expert opinions are consulted to ensure that the restored model conforms to the original historical appearance and artistic style of the relics.
[0087] Texture mapping and rendering: The collected texture data of cultural relics is mapped onto the cultural relic restoration model, so that its surface presents a similar texture and color to the original.
[0088] Multiple solutions comparison and optimization: For the virtual restoration of the same cultural relic, multiple different restoration solutions can be developed, and then these solutions can be displayed simultaneously through augmented reality (AR) technology for comparative analysis.
[0089] Virtual labels and annotations: In AR applications, when a user points their device at a cultural relic, virtual information labels and annotations will automatically appear around or above the relic, introducing the basic information and historical background of the relic in the form of text, pictures, charts and other formats.
[0090] Gesture recognition can be implemented using the corresponding MRTK gesture interaction plugin, which uses the AR glasses' RGB camera and depth camera to capture the position and posture of the user's hand in three-dimensional space.
[0091] Tracking technology is the ability of an image or object to follow the viewer's perspective, keeping it always directly in front of the viewer.
[0092] The execution steps of the cultural relic revitalization and restoration module include:
[0093] Step 11: Acquire images of cultural relics. After capturing images of cultural relics with the RGB camera of the AR glasses, the images are acquired. Image contour segmentation is performed on the acquired images to separate the cultural relics, and then noise reduction is performed to improve the recognition accuracy.
[0094] Step 12: After identifying the cultural relic image after denoising, the missing parts of the cultural relic are restored using modeling software to create a 3D model. At the same time, texture maps are extracted from the collected cultural relic images to fill in the missing textures, resulting in a 3D model of the cultural relic.
[0095] Step 13: Upload the 3D model of the cultural relic to the Unity engine for model manipulation and processing. Develop C# programming functions for gesture interaction or display of the 3D model of the cultural relic.
[0096] Step 14: After the Unity engine is complete, package and install the 3D model of the cultural relic into the AR glasses for use in the APP.
[0097] 2. Object Recognition Module: Utilizing machine learning or deep learning models (especially convolutional neural networks), the model is trained using a training set to enable it to recognize objects in images. The AR glasses transmit real-time data from the real world to the built-in object recognition model for identification and feedback, providing detailed explanations and overlaying them onto the AR glasses.
[0098] The execution steps of the object recognition module include:
[0099] Step 21: Train the entity recognition model. Select the YOLO series model, which belongs to convolutional neural networks, as the initial model. Train the YOLO series model using the training set to obtain the entity recognition model. The sample cultural relic images in the training set must have clear target objects and include various differentiated scenes to ensure data diversity. Label the sample cultural relic images, train the YOLO series model, and optimize and fine-tune the model or change the model type based on the training results to ensure the model's accuracy and recall. Adjust the model architecture according to the actual use scenario.
[0100] Step 22: Actual data collection. The AR glasses' camera collects images of the cultural relics on site and uploads the images to the cloud server.
[0101] Step 23: Actual data processing. The cloud server preprocesses the cultural relic images, and obtains clear images of cultural relics through data cleaning and noise reduction. Feature data is obtained through feature point extraction and entity recognition models.
[0102] Step 24: Model classification and matching. The extracted feature data is compared with the pre-built database to analyze the corresponding matching information, and the corresponding information is transmitted to the engine for further processing.
[0103] Step 25: Model update. Based on the data in use and the increase of actual cultural relic data in the later stage, continuously iterate the entity recognition model, expand the database size, and ensure that the entity recognition model can continue to accurately identify cultural relics.
[0104] Step 26: Send the identified feature data to the Unity program via UDP network communication for data processing. For example, after identifying the name of the model, the engine retrieves the restored cultural relic model and overlays it on the identified cultural relic location to create a timely restoration effect. At the same time, the introduction information of the cultural relic is displayed in the AR glasses.
[0105] 3. Intelligent Character Interaction Module: After the program is launched, a wave of data is collected and the collected information is uploaded to AI. Based on the audience's interests and behaviors, different content is recommended in real time. AI voice narration and dynamic annotation make the exhibition more interactive and personalized.
[0106] The intelligent character interaction module involves the following:
[0107] (1) User data collection: First, user data such as age, gender, geographical location, occupation, etc. can be collected. This information can initially outline the user's profile and bind the corresponding AR glasses.
[0108] (2) Data preprocessing: Cleaning and standardizing the collected user data and content data. For user data, invalid data is removed.
[0109] (3) User profile construction:
[0110] Feature extraction: Extracting features from user data that represent user interests. User profile representation: User profiles can be represented in various ways, such as vectors, labels, or hierarchical structures. Vector representation converts user interest features into numerical vectors, where each dimension represents an interest feature, and the vector value represents the user's degree of interest in that feature.
[0111] (4) Application of recommendation algorithms:
[0112] 1) Content-based recommendation: Calculate the similarity between user profiles (i.e., features representing user interests) and content profiles. Metrics such as cosine similarity and Euclidean distance can be used. Content is ranked according to similarity, and content with high similarity is recommended to the user.
[0113] 2) Collaborative filtering recommendation: Based on user behavior data (i.e., features representing user interests), it finds other users whose behavior is similar to that of the target user. The cold-processing problem of recommendation models can be alleviated by calculating the similarity between users.
[0114] 3) Hybrid Recommendation: Combining the advantages of content-based recommendation and collaborative filtering, different weights are assigned, and the matching degree between content and user interests, as well as the influence of other user behaviors, is comprehensively considered to generate a final recommendation list. This improves the accuracy and diversity of recommendations and avoids the limitations of a single recommendation method.
[0115] 4) Optimization and Adjustment: Based on the evaluation results and user feedback, optimize the recommendation algorithm. Adjust the parameters of the recommendation algorithm, update user profiles and content profiles, and continuously improve the performance of the recommendation system.
[0116] The execution steps of the intelligent character interaction module include:
[0117] Step 31: After startup, the central control unit collects user data and then sends the collected data to the algorithm model server for data analysis via UDP network communication.
[0118] Step 32: The data analyzed by the algorithm model is sent back to the central control unit and the AR glasses program, and content that the user is interested in is recommended in real time to the user participating in the interaction.
[0119] Step 33: Add an AI guide model (3D modeled in 3DMAX) to the AR glasses using the Unity program. This guide will follow the user throughout the interactive route, and each interactive point will be linked to guide the user to interact and provide audio introductions of the cultural relics.
[0120] Step 34: Based on the different basic data of each user, the algorithm model recommends different interactive content in real time, such as particle effects that appear in the interaction, or by superimposing different interactive data with the user's basic data, the algorithm creates some images or model files to be presented in the virtual space.
[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0122] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the system and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. An AR-based system for the restoration and real-time interaction of cultural relics, characterized in that, The AR-based cultural relic restoration and real-time interactive system includes: AR glasses, and a cultural relic activation and restoration module, a physical object recognition module, an intelligent character interaction module, and a database deployed in the AR glasses; the cultural relic activation and restoration module and the physical object recognition module are respectively connected to the database; The cultural relic activation and restoration module is used for: Acquire images of cultural relics captured by the camera of AR glasses; Based on the image of the cultural relic, missing parts are repaired and three-dimensional modeling is performed to obtain a three-dimensional model of the cultural relic. The three-dimensional model of the cultural relic is displayed to the user through the AR glasses; The database is used to store the three-dimensional models of each of the cultural relics and the corresponding cultural relic information; The object recognition module is used for: Acquire images of cultural relics captured by the camera of AR glasses; Using the built-in entity recognition model, the feature data of the cultural relic are determined based on the image of the cultural relic; Based on the feature data, the corresponding 3D model of the cultural relic and the cultural relic information stored in the database are determined. The three-dimensional model of the cultural relic and its information are displayed to the user through the AR glasses. The intelligent character interaction module is used for: Follow the user's route and obtain user data in real time; Based on the user data, the system will recommend audio introductions and interactive content about cultural relics to users in real time.
2. The AR-based cultural relic restoration and real-time interactive system according to claim 1, characterized in that, The AR-based cultural relic restoration and real-time interactive system further includes: a central control unit; the central control unit is connected to the intelligent character interaction module; The central control unit is used to collect the user data.
3. The AR-based cultural relic restoration and real-time interactive system according to claim 1, characterized in that, Based on the image of the cultural relic, missing parts are repaired and 3D modeling is performed to obtain a 3D model of the cultural relic, including: The image of the cultural relic is segmented to obtain a segmented image of the cultural relic. The segmented cultural relic image is denoised to obtain the denoised cultural relic image; Using the built-in modeling software, the missing parts of the cultural relics are repaired and three-dimensionally modeled based on the denoised images of the cultural relics, resulting in a cultural relic restoration model. Determine the texture map based on the image of the cultural relic; The three-dimensional model of the cultural relic is determined based on the cultural relic restoration model and the texture map.
4. The AR-based cultural relic restoration and real-time interactive system according to claim 3, characterized in that, The modeling software includes 3ds Max and Maya.
5. The AR-based cultural relic restoration and real-time interactive system according to claim 1, characterized in that, Using a built-in entity recognition model, based on the image of the cultural relic, the characteristic data of the cultural relic are determined, including: The image of the cultural relic is cleaned and denoised to obtain the processed image of the cultural relic; The processed images of cultural relics are input into the entity recognition model to obtain the feature data of the cultural relics.
6. The AR-based cultural relic restoration and real-time interactive system according to claim 5, characterized in that, The process of determining the entity recognition model includes: Obtain a training set; the training set includes: multiple sample artifact images and corresponding artifact feature data; Construct a convolutional neural network; The convolutional neural network is trained using the training set to obtain the entity recognition model.
7. The AR-based cultural relic restoration and real-time interactive system according to claim 6, characterized in that, The convolutional neural network is trained using the training set to obtain the entity recognition model, comprising: The convolutional neural network is trained using images of each sample cultural relic as input and the corresponding feature data of the cultural relic as output to obtain the entity recognition model.
8. The AR-based cultural relic restoration and real-time interactive system according to claim 6, characterized in that, The entity recognition model is optimized and updated in real time using the expanded training set.
9. The AR-based cultural relic restoration and real-time interactive system according to claim 1, characterized in that, Based on the user data, the system recommends real-time audio descriptions and interactive content about cultural relics to users, including: Using a recommendation algorithm, the system recommends audio descriptions and interactive content about cultural relics to users in real time based on the user data.
10. The AR-based cultural relic restoration and real-time interactive system according to claim 9, characterized in that, The recommendation algorithms include: content-based recommendation, collaborative filtering recommendation, and hybrid recommendation.