Historical scene virtual restoration system and method based on AR and AI

Abstract

The invention provides a historical scene virtual restoration system and method based on AR and AI. The historical scene virtual restoration system based on AR and AI comprises a scene environment AR perception and feature recognition module used for acquiring three-dimensional feature data and environment perception parameters of a target scene in real time through an AR display device and a sensor assembly worn by a user; associating a historical event data loading module; the dynamic historical scene AI generation and optimization module is used for receiving the three-dimensional feature data, the historical event data set and the environment perception parameters and realizing real-time fusion display of a dynamic historical scene and an actual target scene through an AR display device; and the historical scene multi-modal interaction and feedback module is used for realizing multi-modal interaction of the historical scene through the interaction instruction. According to the method, the problems of insufficient dynamic generation capability, space-time experience splitting, single interaction dimension and the like in historical scene restoration are solved, and the method is suitable for cultural heritage protection, text travel experience and historical education scenes.

Description

Technical Field

[0001] The present invention relates to the intersection of augmented reality (AR) and artificial intelligence (AI) technologies, and specifically to a system and method for virtual restoration of historical scenes based on multimodal interaction, which is suitable for cultural heritage protection, cultural tourism experience, and historical education scenarios. Background Art

[0002] In today's rapidly changing technology, augmented reality (AR) and artificial intelligence (AI) are two highly anticipated technology fields. Although they overlap in some aspects, their definitions, applications and working principles are significantly different. Augmented reality (AR) is a technology that enhances the user's perceptual experience by superimposing computer-generated images, videos or other information on the view of the real world. Users can experience AR content through devices such as smartphones, tablets, head-mounted displays, etc. Artificial intelligence (AI) refers to intelligent behavior exhibited by computer systems that can simulate human thinking processes and learning abilities. AI covers multiple subfields such as machine learning, deep learning, and natural language processing, and aims to enable machines to have a certain degree of autonomous decision-making capabilities. Although AR and AI can complement and promote each other's development in some aspects, they each have unique advantages and limitations. Therefore, in practical applications, it is necessary to select appropriate technical solutions based on specific needs. At present, AR and AI applications in existing technologies have the following defects:

[0003] First, there is the technical fragmentation. Although a single AR technology supports environmental capture, it can only display preset 3D models and cannot dynamically generate multi-period historical scenes based on the real-time environment. A single AI image restoration technology cannot generate multi-modal content that is continuous in time and space. The two fail to establish a closed-loop feedback mechanism. Second, there are interactive limitations. Traditional panoramic conversion technology lacks the temporal dimension of historical events, and users cannot switch across time and space scenes. Third, there is data isolation. Existing solutions fail to establish a three-dimensional correlation database of geographic location-timeline-historical events, and fail to achieve the coordinated optimization of AR environmental perception, AI dynamic generation, and multi-modal interaction. Therefore, there is an urgent need for an application in the cross-field of augmented reality (AR) and artificial intelligence (AI) technology, so that it can be applied to various uses such as cultural heritage protection, cultural tourism experience, and historical education scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a historical scene virtual restoration system and method based on AR and AI, which can solve the problems of insufficient dynamic generation capability, fragmented time and space experience, and single interactive dimension in historical scene restoration.

[0005] In order to solve the above problems, the present invention is implemented by adopting the following technical solutions.

[0006] In the first aspect, a historical scene virtual restoration system based on AR and AI is provided, which includes:

[0007] The scene environment AR perception and feature recognition module is used to capture the real scene environment of the target scene in real time through the AR display device worn by the user, display the virtual historical scene, and obtain the user's perspective in real space and the relative position information of the target scene; the sensor component obtains the three-dimensional feature data and environmental perception parameters of the target scene in real time, and obtains the historical spatiotemporal feature information of the target scene by matching the three-dimensional feature data with the architectural feature library;

[0008] An associated historical event data loading module is used to extract and load a historical event data set associated with the historical spatiotemporal feature information from a historical event database using a spatiotemporal relationship matching algorithm;

[0009] A dynamic historical scene AI generation and optimization module is configured to receive the three-dimensional feature data, the historical event data set, and the environmental perception parameters; based on the three-dimensional feature data, perform real-time spatial calculations of the target scene through edge computing nodes and a spatial positioning engine; use a spatiotemporal conflict detector to filter historical event data from the historical event data set that does not match the target spatiotemporal background; and input the filtered historical event data set and the environmental perception parameters into a cloud-based AI server to generate and optimize a virtual dynamic historical scene; and use an AR display device to achieve real-time fusion display of the dynamic historical scene and the actual target scene;

[0010] The historical scene multimodal interaction and feedback module is used to receive interaction instructions through the interactive device to realize multimodal interaction of historical scenes, including historical scene switching based on gesture recognition, multilingual annotation display based on eye tracking, and holographic projection of historical figures based on voice questions. At the same time, the feedback device simultaneously provides tactile and olfactory feedback.

[0011] Furthermore, the AR display device is smart glasses, a head-mounted display or a mobile device; the sensor component includes a depth camera, an IMU sensor (inertial measurement unit), a GPS module, an ambient light sensor, a temperature and humidity sensor, and a PPG (photoplethysmography) sensor; wherein the ambient light sensor can detect ambient light parameters in the 400-1000nm band, and the temperature and humidity sensor has a measurement accuracy of ±0.5°C; the environmental perception parameters specifically include the ambient light parameters, temperature and humidity parameters of the target scene, and user physiological perception data including user heart rate and facial micro-expressions.

[0012] Furthermore, the architectural feature library contains three-dimensional models, architectural styles and historical information of historical buildings, which is used to match the three-dimensional models of the historical buildings with the three-dimensional feature data of the target scene, and extract the architectural style and historical information of the historical buildings corresponding to the target scene; wherein the historical information specifically includes historical time and historical location information associated with the historical buildings.

[0013] Furthermore, the historical event database is a Neo4j graph database, which includes historical event node data and historical event spatiotemporal relationship data, and is used to extract historical event datasets that match the historical spatiotemporal feature information of the target scene through a spatiotemporal relationship matching algorithm and the Cypher query language; wherein the historical event node data stores the corresponding historical period, historical location, historical event description, participant information and historical background attribute information; and the historical event spatiotemporal relationship data stores the spatiotemporal correlation relationship information between each historical event node;

[0014] The spatiotemporal relationship matching algorithm specifically includes: receiving the historical spatiotemporal feature information of the target scene, including historical time information (inputTimePeriod) and historical location information (inputLocation), and matching the historical period attribute (timePeriod) and historical location attribute (location) of the historical event node e in the historical event database through Cypher query. The rule syntax of the Cypher query is as follows:

[0015] MATCH(e:Event)

[0016] WHERE e.timePeriod=$inputTimePeriod AND e.location=$inputLocation

[0017] RETURN

[0018] The matching results are outputted by the Cypher query, and a historical event dataset related to the target scenario is finally returned, including the historical period, historical location, historical event description, participant information and historical background attribute information of the historical event.

[0019] Furthermore, the edge computing node is equipped with the NVIDIA Jetson AGX Orin chip to provide fast data transmission processing, computing and storage services; the spatial positioning engine uses the SLAM algorithm based on the three-dimensional feature data to achieve a positioning accuracy of <2cm, ensuring that users can obtain high-precision spatial positioning and virtual modeling in dynamic scenes; the spatiotemporal conflict detector specifically detects the historical period and historical location attribute information in the historical event dataset, screens out historical event nodes and historical event spatiotemporal relationship data that do not match the target spatiotemporal background, and extracts and loads the filtered historical event dataset.

[0020] Furthermore, the cloud AI server includes a dynamic rendering engine and an emotional computing component, wherein the dynamic rendering engine is used to output a dynamic historical scene sequence frame based on the input historical event data set and the ambient light parameters and temperature and humidity parameters of the target scene in the environmental perception parameters, generate a virtual dynamic historical scene, and dynamically adjust the level of detail of objects in the scene according to the distance and viewing angle of the user from the scene through a dynamic LOD algorithm to optimize rendering performance and enhance user experience; the emotional computing component adjusts the expressiveness of visual and auditory elements in the scene based on the input user physiological perception data to generate a personalized historical scene experience;

[0021] The dynamic LOD algorithm calculates the optimal LOD level and automatically loads the three-dimensional model with the corresponding level of detail by real-time monitoring of the user's perspective in real space and position information relative to the target scene obtained through the AR display device. By combining it with GPU accelerated rendering, it can reduce resource consumption and improve rendering efficiency while ensuring the generation of the visual effect of the scene, ensuring the smooth operation of the system on mobile devices.

[0022] Furthermore, the interaction device includes a gesture recognition camera, an eye tracking device and a microphone, which are used to identify interaction instructions including user gestures, detect and track target scenes, and obtain user voice; the feedback device specifically includes tactile gloves and a micro-odor generator, which are used to synchronously provide tactile feedback and olfactory feedback of historical scenes based on the interaction instructions; the tactile gloves have 20 built-in vibration motors that can simulate the touch of materials in the range of 0-1N; the micro-odor generator has an odor feedback response delay of <50ms, ensuring that users can perceive olfactory feedback related to historical scenes in a timely manner during the interaction process to enhance the immersive experience.

[0023] In addition, the present invention also discloses a method for virtual restoration of historical scenes based on AR and AI, comprising the following steps:

[0024] Step 1: Acquire the three-dimensional feature data and environmental perception parameters of the target scene through the AR display device and sensor components worn by the user;

[0025] Step 2: Match the three-dimensional feature data with the building feature library to obtain historical spatiotemporal feature information of the target scene;

[0026] Step 3: extracting a historical event dataset related to the historical spatiotemporal feature information from a historical event database;

[0027] Step 4: Perform real-time spatial calculations on the three-dimensional feature data, historical event dataset, and environmental perception parameters through edge computing nodes and spatial positioning engines, and use a spatiotemporal conflict detector to filter out historical event data that does not match the target spatiotemporal context.

[0028] Step 5: Input the filtered historical event data set and the environmental perception parameters into the cloud AI server to generate a virtual dynamic historical scene, and realize the real-time fusion display of the dynamic historical scene and the actual target scene through the AR display device;

[0029] Step 6: Implement interaction and feedback between users and historical scenarios through the multimodal interaction and feedback module.

[0030] Furthermore, in step 5, the cloud AI server generates a dynamic historical scene based on the input historical event data set, and adjusts and optimizes the visual and auditory expressiveness of the scene according to the user's environmental perception parameters to generate a personalized historical scene experience.

[0031] Furthermore, the interaction in step 6 specifically includes recognizing a preset gesture through a gesture recognition camera and switching the historical scene to the target era version, tracking and detecting the target area gazed by the user with the help of an eye tracking device and displaying multi-language annotations, receiving voice questions through a microphone and generating a holographic projection of historical figures; the feedback in step 6 specifically includes synchronously providing tactile feedback and olfactory feedback through a multimodal feedback device consisting of a tactile glove and a micro odor generator to enhance the user's immersive experience.

[0032] Furthermore, the method can be widely used in fields such as education, tourism, and cultural heritage protection to enhance users' understanding and experience of history.

[0033] By means of the above technical solution, the present invention has the following technical effects:

[0034] 1) This AR and AI-based historical scene virtual restoration system uses a scene environment AR perception and feature recognition module to extract the historical spatiotemporal characteristics of the target scene in real time. Combined with a dynamic historical scene AI generation and optimization module, it enables dynamic reconstruction of multi-period historical scenes. Compared to traditional static model restoration solutions, this system can accurately restore historical scenes in real time, creating a personalized virtual historical scene experience that is continuous in time and space.

[0035] 2) The present invention's historical scene virtual restoration system based on AR and AI stores and extracts historical event information with spatiotemporal correlation based on the Neo4j graph database. Through dual verification by the spatial positioning engine and the spatiotemporal conflict detector, it can accurately load dynamic historical scenes and ensure that the spatial coordinates of the virtual historical scenes are accurately aligned with the real space environment. The system solves the problems of misalignment of virtual and real scenes and confusion in the spatiotemporal logic of historical events in traditional solutions.

[0036] 3) This AR and AI-based method for virtualizing historical scenes utilizes a multimodal interaction and feedback module for historical scenes, integrating gesture recognition, eye tracking, voice control, and tactile and olfactory feedback. This allows users to switch between historical scenes, trigger multilingual annotations, or engage in holographic dialogues with historical figures. This multimodal interaction extends the interaction dimension from a single visual perspective to a multimodal "sight-hearing-touch-smell" experience, enhancing user immersion and interactive experience, and providing new approaches and means for historical education and cultural communication.

[0037] 4) The virtual restoration method of historical scenes based on AR and AI in the present invention can efficiently and quickly complete computing tasks such as spatial positioning and spatiotemporal conflict detection through collaborative processing of edge computing and cloud-based AI servers. Combined with the real-time analysis of user physiological data by emotional computing components, it can better restore and display dynamic, realistic and personalized historical scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the system structure for virtual restoration of historical scenes based on AR and AI.

[0039] Figure 2 This is a flowchart for executing the spatiotemporal relationship matching algorithm; it shows the complete process of extracting a historical event dataset related to the historical spatiotemporal feature information of the target scene from the historical event database using the Cypher query language.

[0040] Figure 3 Flowchart for the dynamic LOD algorithm; it shows the complete process of dynamically adjusting the level of detail of objects in the virtual historical scene based on the user's distance from the target scene and the viewing angle.

[0041] Figure 4 A flowchart for triggering the virtual restoration of historical scenes based on AR and AI; the diagram shows the complete steps of virtual restoration of historical scenes based on AR and AI.

[0042] Figure 5 User interaction diagram; the diagram shows examples of users interacting with historical scenes through gestures, eye movements, and voice.

[0043] Figure 6This is a workflow diagram of the virtual restoration system of historical scenes based on AR and AI; the diagram shows the complete workflow from user environment perception to virtual restoration of historical scenes, including each link of data acquisition, processing, generation and feedback. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below by using specific preferred embodiments in conjunction with the accompanying drawings, but the present invention is not limited to the following embodiments. The following specific embodiments and drawings can more clearly understand the structure, function and application scenarios of the present invention.

[0045] like Figure 1 As shown, the present invention discloses a historical scene virtual restoration system based on AR and AI, including a scene environment AR perception and feature recognition module 1, an associated historical event data loading module 2, a dynamic historical scene AI generation and optimization module 3, and a historical scene multimodal interaction and feedback module 4.

[0046] The scene environment AR perception and feature recognition module 1 is used to capture the real scene environment of the target scene in real time through the AR display device 10 worn by the user and display the virtual historical scene, and obtain the user's perspective in real space and the relative position information with the target scene; the sensor component 11 is used to obtain the three-dimensional feature data and environmental perception parameters of the target scene in real time, and the historical spatiotemporal feature information of the target scene is obtained by matching the three-dimensional feature data with the architectural feature library 12;

[0047] The associated historical event data loading module 2 is used to extract and load the historical event data set associated with the historical spatiotemporal feature information from the historical event database 20 through a spatiotemporal relationship matching algorithm;

[0048] The dynamic historical scene AI generation and optimization module 3 is used to receive the three-dimensional feature data, the historical event data set and the environmental perception parameters, and based on the three-dimensional feature data, realize real-time spatial calculation of the target scene through the edge computing node 30 and the spatial positioning engine 31, and use the spatiotemporal conflict detector 32 to filter the historical event data that does not match the target spatiotemporal background, and input the filtered historical event data set and the environmental perception parameters into the cloud AI server 33 to realize the generation and optimization of the virtual dynamic historical scene, and realize the real-time fusion display of the dynamic historical scene and the actual target scene through the AR display device 10;

[0049] The historical scene multimodal interaction and feedback module 4 is used to receive interaction instructions through the interaction device 40 to realize multimodal interaction of historical scenes, including historical scene switching based on gesture recognition, multilingual annotation display based on eye tracking, and holographic projection of historical figures based on voice questions. At the same time, the feedback device 41 synchronously provides tactile and olfactory feedback.

[0050] Furthermore, each module of the system is composed of the following parts:

[0051] AR display device 10: This device can be smart glasses (such as Microsoft HoloLens), a head-mounted display, or a mobile device (such as a smartphone), connected to the mobile device via Bluetooth or Wi-Fi, and is used to capture the real scene environment of the target scene in real time, obtain the user's perspective in real space and the relative position information of the target scene, receive data, and display an interface for the user to interact with the virtual historical scene;

[0052] Sensor assembly 11: includes a depth camera, an IMU sensor (inertial measurement unit), a GPS module, an ambient light sensor, a temperature and humidity sensor, and a PPG (photoplethysmography) sensor. Each of the sensors is connected to the AR display device 10 and is used to obtain real-time environmental information of the target scene, identify and obtain three-dimensional feature data and environmental perception parameters of the user's location; wherein the ambient light sensor can detect ambient light parameters in the 400-1000nm band, and the temperature and humidity sensor has a precision of ±0.5°C; the environmental perception parameters specifically include the ambient light parameters, temperature and humidity parameters of the target scene, and user physiological perception data including user heart rate and facial micro-expressions;

[0053] Architectural feature library 12: uses cloud storage technology to store the three-dimensional models, architectural styles, and historical information of historical buildings. It receives the three-dimensional data of the target scene acquired by the sensor component 11, and is used to extract the architectural style and historical information of the historical buildings corresponding to the target scene, obtain the historical spatiotemporal feature information of the target scene, and connect to the associated historical event dataset loading module through the API interface;

[0054] The historical event database 20 is a Neo4j graph database that uses cloud storage technology to store historical event node data and historical event spatiotemporal relationship data. It is connected to the dynamic historical scene AI generation and optimization module through an API interface, receives the historical spatiotemporal feature information of the target scene obtained by the building feature library 12, and extracts and loads matching historical event datasets with the help of Cypher queries through a spatiotemporal relationship matching algorithm. The historical event nodes include the corresponding historical period, historical location, historical event description, participant information, and historical background attribute information. The historical event spatiotemporal relationship data includes the spatiotemporal association relationship information between each historical event node.

[0055] Edge computing node 30: equipped with an NVIDIA Jetson AGX Orin chip, used to receive the three-dimensional feature data, environmental perception parameters, and historical event data sets loaded from the historical event database 20 from the sensor component 11, and provide fast data transmission processing, computing, and storage services;

[0056] Spatial Positioning Engine 31: Uses SLAM algorithm to achieve positioning accuracy of <2cm, ensuring users obtain high-precision spatial positioning and virtual modeling in dynamic scenes;

[0057] Spatiotemporal conflict detector 32: used to detect the historical period and historical location attribute information in the historical event dataset, filter out historical event nodes and historical event spatiotemporal relationship data that do not match the target spatiotemporal background, and extract and load the filtered historical event dataset;

[0058] The cloud AI server 33 includes a dynamic rendering engine and an emotional computing component, which is configured to output a dynamic historical scene sequence frame based on the input historical event data set and the ambient light parameters and temperature and humidity parameters of the target scene in the environmental perception parameters through the dynamic rendering engine, thereby generating a virtual dynamic historical scene. The server also dynamically adjusts the level of detail of objects in the scene based on the distance and viewing angle of the user from the scene through a dynamic LOD algorithm to optimize rendering performance, further enhance the expressiveness of the virtual historical scene, and improve the user experience. The emotional computing component adjusts the expressiveness of visual and auditory elements in the scene based on the input user physiological perception data to generate a personalized historical scene experience, and transmits the virtual dynamic historical scene data to the AR display device 10.

[0059] Interaction device 40: includes a gesture recognition camera, an eye tracking device, and a microphone, connected to the AR display device 10, and is used to recognize interaction instructions including user gestures, detect and track target scenes, and obtain user voice;

[0060] Feedback device 41: includes a tactile glove and a micro-odor generator, connected to the AR display device 10, and is used to synchronously provide tactile feedback and olfactory feedback of the historical scene based on the interactive instructions. The tactile glove has 20 built-in vibration motors that can simulate the touch of materials in the range of 0-1N. The micro-odor generator has an odor feedback response delay of <50ms, ensuring that the user can perceive the olfactory feedback related to the historical scene in a timely manner during the interaction process to enhance the immersive experience.

[0061] Further, such as Figure 2 The figure shows the execution flow chart of the spatiotemporal relationship matching algorithm. It shows the complete process of extracting the historical event dataset related to the historical spatiotemporal feature information of the target scene from the historical event database using the Cypher query language. The specific process is as follows:

[0062] Receive the target scene's historical spatiotemporal feature information input into the associated historical event data loading module, load historical event nodes from the historical event database; execute Cypher query statements, and when a historical event node matches the target scene's historical spatiotemporal feature information, extract the node, and extract historical event nodes associated with the node based on the historical event spatiotemporal relationship in the historical event data, continue to load historical event nodes in sequence, repeat the above process until all historical event nodes are loaded, and output the historical event data set related to the target scene; when a historical event node does not match the target scene's historical spatiotemporal feature information, continue to load historical event nodes in sequence and repeat the above process; wherein the rule syntax of the Cypher query statement is as follows:

[0063] MATCH(e:Event)

[0064] WHERE e.timePeriod=$inputTimePeriod AND e.location=$inputLocation

[0065] RETURN

[0066] Among them, e is a historical event node. By judging the matching relationship between the historical period attribute (timePeriod) and historical location attribute (location) of the historical event node e and the historical time information (inputTimePeriod) and historical location information (inputLocation) in the historical spatiotemporal feature information of the target scene, the matching historical event node is returned. The historical node contains the corresponding historical period, historical location, historical event description, participant information and its historical background information.

[0067] Further, such as Figure 3 The figure shows the flow chart of the dynamic LOD algorithm execution. It shows the complete process of dynamically adjusting the level of detail of objects in the virtual historical scene based on the distance and viewing angle between the user and the target scene. The specific process is as follows:

[0068] The user's viewing angle α in the real space and the relative distance d to the specific object in the target scene are obtained in real time through the AR display device. When the conditions of user viewing angle α < 5° and relative distance d < 5m are met at the same time, a high-detail model is automatically loaded, and combined with GPU accelerated rendering, the material texture and structural form of the target scene are refined to generate a high-detail dynamic virtual historical scene. When the user is away from the target scene to the point where the conditions are no longer met, the model is switched to a low-detail model to reduce resource consumption and improve rendering efficiency while ensuring the visual effect of generating a dynamic virtual historical scene.

[0069] In addition, if Figure 4 The figure shows a flowchart of the steps of the virtual restoration method of historical scenes based on AR and AI of the present invention. The virtual restoration method of historical scenes based on AR and AI of the present invention includes the following steps:

[0070] Step S1: Acquire three-dimensional feature data and environmental perception parameters of a target scene through an AR display device and a sensor component worn by a user;

[0071] Step S2: matching the three-dimensional feature data with the building feature library to obtain historical spatiotemporal feature information of the target scene;

[0072] Step S3: extracting a historical event data set related to the historical spatiotemporal feature information from a historical event database;

[0073] Step S4: performing real-time spatial calculations on the three-dimensional feature data, historical event data set, and environmental perception parameters through edge computing nodes and a spatial positioning engine, and using a spatiotemporal conflict detector to filter historical event data that does not match the target spatiotemporal context;

[0074] Step S5: Inputting the filtered historical event dataset and the environmental perception parameters into a cloud-based AI server to generate a virtual dynamic historical scene, and realizing real-time fusion display of the dynamic historical scene and the actual target scene through an AR display device; the cloud-based AI server generates a dynamic historical scene based on the input historical event dataset, and adjusts and optimizes the visual and auditory expressiveness of the scene based on the user's environmental perception parameters to generate a personalized historical scene experience;

[0075] Step S6: Implementing interaction and feedback between the user and the historical scene through a multimodal interaction and feedback module; the interaction specifically includes recognizing preset gestures and switching the historical scene to the target era version, eye tracking detecting the target area the user is gazing at and displaying multilingual annotations, receiving voice questions and generating holographic projections of historical figures; the feedback specifically includes providing tactile feedback and olfactory feedback through a multimodal feedback device consisting of a tactile glove and a micro odor generator to enhance the user's immersive experience.

[0076] Further, such as Figure 5 The figure shown is an example of a user interacting with a historical scene through an interactive device and a feedback device with the help of interactive instructions. Figure 3The gesture recognition camera, eye tracker, and microphone together form the interaction device, which is used to identify the user's multimodal interaction commands. When a user performs a preset gesture, the gesture recognition camera captures the action and converts it into a command, triggering the system to switch to the historical scene version of the target period. The AR device updates the virtual scene in real time and overlays it with the real environment. When the user gazes at a specific area, the eye tracker locates the gaze point through infrared detection, extracts related historical event data, and displays floating multilingual annotations. After the user's voice question is analyzed by natural language in the cloud, a dynamic rendering engine generates a holographic projection of the historical figure, synthesizes the voice answer, and displays subtitles simultaneously, which are overlaid on the relevant location in the scene through the AR display device. In addition, a tactile glove and a micro-odor generator together form a feedback device, which is used to provide users with simultaneous tactile and olfactory feedback. The tactile glove simulates the physical touch of the scene through micro-vibrations, and the micro-odor generator releases temporally and spatially matched odors, and the diffusion module controls the intensity and duration of the feedback.

[0077] The method of the present invention can be widely used in fields such as education, tourism, and cultural heritage protection to enhance users' understanding and experience of history.

[0078] Figure 6 This is the workflow diagram of the historical scene virtual restoration system based on AR and AI in this invention, which includes four parts: system architecture, data loading, dynamic generation, and interaction and feedback. Specifically, it includes:

[0079] System Architecture: The user uses the AR display device 10 and the sensor assembly 11 to obtain the target scene's three-dimensional feature data, environmental perception parameters, and the user's perspective in real space and relative position information to the target scene in real time. The user then matches the three-dimensional feature data with the building feature library 12 to obtain historical spatiotemporal feature information.

[0080] Data loading: receiving the historical spatiotemporal feature information of the target scene, extracting and loading the historical event dataset associated with the input historical spatiotemporal feature information from the historical event database 20 through the spatiotemporal relationship matching algorithm;

[0081] Dynamic generation: Receives associated historical event data sets, three-dimensional feature data, and environmental perception parameters, performs preliminary processing and calculation on the input data through the edge computing node 30, combines with the spatial positioning engine 31 to perform real-time spatial calculations, and the spatiotemporal conflict detector 32 filters historical event data that does not match the target spatiotemporal background; the cloud AI server 33 receives the filtered historical event data and environmental perception parameters, generates scene sequence frames through the dynamic rendering engine therein, and uses the dynamic LOD algorithm to dynamically adjust the detail level of objects in the virtual historical scene according to the distance between the user and the target scene, optimizes rendering performance and improves user experience. The emotional computing component further adjusts and optimizes the expressiveness of the virtual historical scene through the user's physiological data in the environmental perception parameters to form a personalized virtual historical scene. The virtual scene is displayed in a fusion of the virtual historical scene and the actual scene environment through the AR device 10;

[0082] Interaction and feedback: With the help of the interactive device 40, the user recognizes preset gestures and triggers historical scene switching, detects the eye tracking area and triggers the display of multi-language annotations, receives voice questions and generates holographic projections of historical figures, and the feedback device 41 simultaneously provides tactile and olfactory feedback.

[0083] Through collaborative processing and multimodal fusion, combined with dynamic LOD rendering technology, the system can reduce delays and freezes while taking into account the authenticity of the visual effects of historical scenes, and smoothly achieve high-precision historical scene restoration and immersive interactive experience.

[0084] The following are application efficacy test examples of the historical scene virtual restoration system and method based on AR and AI of the present invention.

[0085] Application Example 1: Virtual Restoration of Ancient Towns

[0086] System Architecture: A user wearing an AR display device 10 connected to a sensor assembly 11 enters the actual scene of a historical site. The scene environment AR perception and feature recognition module acquires the site's three-dimensional feature data in real time and matches it with the data in the architectural feature library 12, identifying the site as an ancient town.

[0087] Data loading: The associated historical event data loading module extracts historical event data sets related to the ancient town from the historical event database 20 with the help of a spatiotemporal relationship matching algorithm, such as the establishment of the town, important historical events, etc.

[0088] Dynamic Generation: The dynamic historical scene AI generation and optimization module receives 3D feature data, historical event datasets, and environmental perception parameters. Through edge computing nodes 30 and spatial positioning engine 31, it performs real-time calculations to generate a virtual scene of the ancient town. A spatiotemporal conflict detector 32 filters out event data that doesn't align with the actual historical context, ensuring the generated scene is authentic and believable. A dynamic LOD algorithm enhances the expressiveness of the virtual ancient town scene, reduces resource consumption, and ensures a smoother, more immersive user experience.

[0089] Interactive feedback: Users can use gestures to switch between scenes from different historical periods using interactive device 40. Eye tracking technology provides multilingual annotations, and users can also ask questions to learn about historical figures in the town. The feedback device (41) provides tactile and olfactory feedback, such as simulating the smell of an ancient market and the texture of wall materials, to enhance immersion.

[0090] Application Example 2: Reproduction of Historical Battles

[0091] System Architecture: A user wears an AR display device 10 connected to a sensor assembly 11 and visits the site of a historical battle. The scene environment AR perception and feature recognition module acquires 3D feature data of the battle site in real time.

[0092] Data loading: The associated historical event data loading module extracts the historical event data set related to the battle from the historical event database 20 with the help of the spatiotemporal relationship matching algorithm, including the background of the battle, participants and the course of the battle.

[0093] Dynamic Generation: The AI-powered historical scene generation and optimization module receives 3D feature data, historical event datasets, and environmental perception parameters to generate a virtual battle scene. A spatiotemporal conflict detector 32 ensures that only event data relevant to the battle is displayed. A dynamic LOD algorithm improves the expressiveness of the virtual battle scene, reduces resource consumption, and ensures a smoother, more immersive user experience.

[0094] Interactive Feedback: Users use gestures to navigate through different stages of the battle using interactive device 40. Eye tracking technology displays multilingual annotations of each character in the battle. Users can also ask questions through voice to obtain detailed information about the battle. Feedback device 41 provides feedback on the sounds and smells of the battle, enhancing user engagement.

[0095] Application Example 3: Experience of Ancient Cultural Activities

[0096] System Architecture: A user wears an AR display device 10 connected to a sensor assembly 11 and enters a venue where a historical or cultural event is taking place. The scene environment AR perception and feature recognition module acquires the three-dimensional feature data of the venue in real time.

[0097] Data loading: The associated historical event data loading module extracts historical event datasets related to the cultural activity, such as festivals, ceremonies, etc., from the historical event database 20 with the help of a spatiotemporal relationship matching algorithm.

[0098] Dynamic generation: The dynamic historical scene AI generation and optimization module generates virtual scenes of cultural activities through the edge computing node 30 and the spatial positioning engine 31, and filters out irrelevant event data. It improves the expressiveness of virtual cultural activity scenes through the dynamic LOD algorithm, reduces resource consumption, and ensures that users have a smoother immersive experience.

[0099] Interactive Feedback: Users can use gestures to select different cultural activities using interactive device 40. Eye tracking technology displays multilingual annotations of the activities. Users can also ask voice questions to understand the background and significance of the activities. Multimodal feedback device 41 provides tactile and olfactory feedback related to the activities, enhancing the authenticity of the cultural experience.

[0100] The AR and AI-based historical scene virtual restoration system and method provided by the present invention can restore historical scenes in real time and accurately, enhance the user's immersion and interactive experience, and provide new methods and means for historical education and cultural communication.

[0101] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A historical scene virtual restoration system based on AR and AI, characterized by It includes: The scene environment AR perception and feature recognition module is used to capture the real scene environment of the target scene in real time through the AR display device worn by the user, display the virtual historical scene, and obtain the user's perspective in real space and the relative position information of the target scene; the sensor component obtains the three-dimensional feature data and environmental perception parameters of the target scene in real time, and obtains the historical spatiotemporal feature information of the target scene by matching the three-dimensional feature data with the architectural feature library; An associated historical event data loading module is used to extract and load a historical event data set associated with the historical spatiotemporal feature information from a historical event database using a spatiotemporal relationship matching algorithm; A dynamic historical scene AI generation and optimization module is configured to receive the three-dimensional feature data, the historical event data set, and the environmental perception parameters; based on the three-dimensional feature data, perform real-time spatial calculations of the target scene through edge computing nodes and a spatial positioning engine; use a spatiotemporal conflict detector to filter historical event data from the historical event data set that does not match the target spatiotemporal background; and input the filtered historical event data set and the environmental perception parameters into a cloud-based AI server to generate and optimize a virtual dynamic historical scene; and use an AR display device to achieve real-time fusion display of the dynamic historical scene and the actual target scene; The historical scene multimodal interaction and feedback module is used to receive interaction instructions through the interactive device to realize multimodal interaction of historical scenes, including historical scene switching based on gesture recognition, multilingual annotation display based on eye tracking, and holographic projection of historical figures based on voice questions. At the same time, the feedback device simultaneously provides tactile and olfactory feedback.

2. The system according to claim 1, wherein: The AR display device is smart glasses, a head-mounted display or a mobile device; the sensor component includes a depth camera, an IMU sensor (inertial measurement unit), a GPS module, an ambient light sensor, a temperature and humidity sensor, and a PPG (photoplethysmography) sensor; wherein the ambient light sensor can detect ambient light parameters in the 400-1000nm band, and the temperature and humidity sensor has a measurement accuracy of ±0.5°C; the environmental perception parameters specifically include the ambient light parameters, temperature and humidity parameters of the target scene, and user physiological perception data including user heart rate and facial micro-expressions.

3. The system according to claim 1, wherein: The architectural feature library contains three-dimensional models, architectural styles and historical information of historical buildings, and is used to match the three-dimensional models of the historical buildings with the three-dimensional feature data of the target scene, and extract the architectural style and historical information of the historical buildings corresponding to the target scene; wherein the historical information specifically includes historical time and historical location information associated with the historical buildings.

4. The system according to claim 1, wherein: The historical event database is a Neo4j graph database, which contains historical event node data and historical event spatiotemporal relationship data, and is used to extract historical event datasets that match the historical spatiotemporal feature information of the target scene through a spatiotemporal relationship matching algorithm and the Cypher query language; wherein the historical event node data stores the corresponding historical period, historical location, historical event description, participant information and historical background attribute information; and the historical event spatiotemporal relationship data stores the spatiotemporal correlation relationship information between each historical event node; The spatiotemporal relationship matching algorithm specifically includes: receiving historical spatiotemporal feature information of the target scene, including historical time information and historical location information; matching the historical period attributes and historical location attributes of the historical event node e in the historical event database in sequence through Cypher query; outputting the matching results through the Cypher query; and finally returning a historical event dataset related to the target scene, including the historical period, historical location, historical event description, participant information and historical background attribute information of the historical event.

5. The system according to claim 1, wherein: The edge computing node is equipped with the NVIDIA Jetson AGX Orin chip, which is used to provide fast data transmission processing, computing and storage services; the spatial positioning engine uses the SLAM algorithm based on the three-dimensional feature data to achieve a positioning accuracy of <2cm, ensuring that users can obtain high-precision spatial positioning and virtual modeling in dynamic scenes; the spatiotemporal conflict detector specifically detects the historical period and historical location attribute information in the historical event dataset, screens out historical event nodes and historical event spatiotemporal relationship data that do not match the target spatiotemporal background, and extracts and loads the filtered historical event dataset.

6. The system according to claim 1, wherein: The cloud AI server includes a dynamic rendering engine and an emotional computing component, wherein the dynamic rendering engine is used to output a dynamic historical scene sequence frame based on the input historical event data set and the ambient light parameters and temperature and humidity parameters of the target scene in the environmental perception parameters, generate a virtual dynamic historical scene, and dynamically adjust the level of detail of objects in the scene based on the distance and viewing angle of the user from the scene through a dynamic LOD algorithm to optimize rendering performance and enhance user experience; the emotional computing component adjusts the expressiveness of visual and auditory elements in the scene based on the input user physiological perception data to generate a personalized historical scene experience; The dynamic LOD algorithm calculates the optimal LOD level and automatically loads the three-dimensional model with the corresponding level of detail by real-time monitoring of the user's perspective in real space and position information relative to the target scene obtained through the AR display device. By combining it with GPU accelerated rendering, it can reduce resource consumption and improve rendering efficiency while ensuring the generation of the visual effect of the scene, ensuring the smooth operation of the system on mobile devices.

7. The system according to claim 1, wherein: The interactive device includes a gesture recognition camera, an eye tracking device and a microphone, which are used to identify interactive instructions including user gestures, detect and track target scenes, and obtain user voice; the feedback device specifically includes tactile gloves and a micro-odor generator, which are used to synchronously provide tactile feedback and olfactory feedback of historical scenes based on the interactive instructions; the tactile gloves have 20 built-in vibration motors that can simulate the touch of materials in the range of 0-1N; the micro-odor generator has an odor feedback response delay of <50ms, ensuring that users can perceive olfactory feedback related to historical scenes in a timely manner during the interaction process to enhance the immersive experience.

8. A restoration method using the AR and AI-based historical scene virtual restoration system of claim 1, comprising the following steps: Step 1: Acquire the three-dimensional feature data and environmental perception parameters of the target scene through the AR display device and sensor components worn by the user; Step 2: Match the three-dimensional feature data with the building feature library to obtain historical spatiotemporal feature information of the target scene; Step 3: extracting a historical event dataset related to the historical spatiotemporal feature information from a historical event database; Step 4: Perform real-time spatial calculations on the three-dimensional feature data, historical event dataset, and environmental perception parameters through edge computing nodes and spatial positioning engines, and use a spatiotemporal conflict detector to filter out historical event data that does not match the target spatiotemporal context. Step 5: Input the filtered historical event data set and the environmental perception parameters into the cloud AI server to generate a virtual dynamic historical scene, and realize the real-time fusion display of the dynamic historical scene and the actual target scene through the AR display device; Step 6: Implement interaction and feedback between users and historical scenarios through the multimodal interaction and feedback module.

9. The method according to claim 8, characterized in that: In step 5, the cloud AI server generates a dynamic historical scene based on the input historical event data set, and adjusts and optimizes the visual and auditory expressiveness of the scene according to the user's environmental perception parameters to generate a personalized historical scene experience.

10. The method according to claim 8, characterized in that: The interaction in step 6 specifically includes recognizing a preset gesture through a gesture recognition camera and switching the historical scene to the target era version, tracking and detecting the target area gazed by the user with the help of an eye tracking device and displaying multi-language annotations, receiving voice questions through a microphone and generating a holographic projection of historical figures; the feedback in step 6 specifically includes synchronously providing tactile feedback and olfactory feedback through a multimodal feedback device consisting of a tactile glove and a micro odor generator to enhance the user's immersive experience.

Images