Virtual-real fusion exhibition display interaction system and multi-modal perception method
By combining the multimodal perception module and the data processing module, the virtual exhibition content and the physical exhibition scene are accurately integrated, which solves the problems of single interaction mode and insufficient perception accuracy in the existing technology, and improves the user interaction experience and the personalized service capability of the exhibition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU ART & DESIGN TECH INST
- Filing Date
- 2026-03-26
- Publication Date
- 2026-06-05
AI Technical Summary
Existing virtual-real integrated exhibition systems suffer from limited interaction methods, insufficient perception accuracy, poor multimodal data synchronization, and inadequate feature fusion, making it difficult to accurately capture users' diverse interactive intentions.
It employs a virtual-real fusion display module, a multimodal perception module, an interactive control module, and a data processing module. Through multi-dimensional interactive data collection, feature extraction, and semantic analysis, it achieves precise integration of virtual exhibition content and physical exhibition scenes, as well as multi-channel feedback.
It improves the accuracy of user interaction intent recognition, enhances immersion and real-time interaction response, supports personalized exhibition content recommendations, and improves the immersion and interactivity of exhibitions.
Smart Images
Figure CN122152135A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of exhibition and display technology and human-computer interaction technology, specifically to a virtual-real integrated exhibition and display interaction system and a multimodal perception method. Background Technology
[0002] In the field of cultural dissemination and information display, exhibitions, as an important medium, are transforming towards digitalization, immersion, and interactivity. Traditional physical exhibitions rely on physical spaces to display exhibits, which not only have inherent defects such as fixed content, limited information capacity, and poor geographical accessibility, but also fail to meet the needs of contemporary users for personalized and deeply participatory experiences, resulting in limited exhibition dissemination efficiency and user engagement.
[0003] The rise of virtual-real fusion technologies (such as the combination of AR / VR with physical scenes) has provided an effective path for upgrading exhibitions. By integrating virtual exhibits, dynamic effects, and physical exhibition environments, it breaks the spatial and content limitations of traditional exhibitions and enriches the forms of display. However, current virtual-real fusion systems applied to exhibition scenarios still have significant shortcomings in human-computer interaction: most systems only support single-modal interaction (such as touch and simple voice commands), and cannot fully capture users' visual, motor, and voice-based multi-faceted interactive behaviors, leading to deviations in the understanding of users' interactive intentions; at the same time, the accuracy and real-time performance of interactive responses are insufficient, easily resulting in problems such as interaction delays and feedback misalignments, which seriously affect the user's immersive experience.
[0004] Furthermore, existing multimodal perception technologies have poor adaptability to exhibition scenarios. The synchronization of data collection across different modalities (visual, motion, and speech) is insufficient, and the feature fusion logic is simplistic, failing to effectively integrate multi-dimensional interactive information to support accurate intent recognition. This results in the system's inability to quickly and accurately respond to users' personalized interaction needs. In summary, existing technologies suffer from technical problems such as a single interactive modality in virtual-real fusion exhibition systems, weak multimodal data collaborative perception capabilities, and insufficient accuracy in user intent recognition. These issues hinder the upgrade of the virtual-real fusion exhibition experience. Therefore, there is an urgent need to develop an exhibition display interaction system and method that can achieve accurate fusion of virtual and physical scenes and efficient perception and analysis of multimodal interactive data, thereby enhancing the immersiveness, interactivity, and personalized service capabilities of exhibitions. Summary of the Invention
[0005] The technical problem to be solved by the present invention is that the existing virtual-real fusion exhibition system has a single interaction mode, insufficient perception accuracy, poor multimodal data synchronization and poor feature fusion effect, making it difficult to accurately capture the user's diverse interaction intentions.
[0006] The technical solution adopted in this invention is: a virtual-real integrated exhibition and display interactive system, comprising: a virtual-real integrated display module, a multimodal perception module, an interactive control module, a data processing module, and a storage module;
[0007] The virtual-real fusion display module is used to load preset virtual exhibition content and output the virtual exhibition content and the physical exhibition scene in real time after fusion rendering.
[0008] The multimodal perception module is used to collect multi-dimensional interaction data of users, which includes at least visual interaction data, action interaction data and voice interaction data.
[0009] The data processing module is connected to the multimodal perception module and the storage module respectively, and is used to preprocess, extract features and perform semantic parsing on the multi-dimensional interactive data to obtain the user's interaction intent, and call the exhibition-related data pre-stored in the storage module.
[0010] The interactive control module is connected to the data processing module and the virtual-real fusion display module respectively. It is used to generate corresponding display adjustment instructions and interactive feedback instructions according to the user's interaction intention and exhibition-related data, and send them to the virtual-real fusion display module.
[0011] The virtual-real fusion display module responds to the display adjustment command and interactive feedback command, adjusts the display status of the virtual exhibition content, and outputs corresponding interactive feedback information.
[0012] As a further aspect of the present invention: the virtual-real fusion display module includes a virtual content loading unit, a scene fusion rendering unit, and a multi-channel output unit; the virtual content loading unit is used to parse and load virtual exhibition models, animations, and audio resources in a preset format; the scene fusion rendering unit is used to obtain spatial parameters of the physical exhibition scene through spatial positioning technology, and to achieve accurate matching between the virtual exhibition content and the physical exhibition scene using a coordinate alignment algorithm, wherein the coordinate alignment formula is: ;in, The three-dimensional coordinates representing the content of the virtual exhibition. Representing the three-dimensional coordinates of a physical exhibition scene. It is a 3×3 rotation matrix used to align the virtual coordinates with the physical coordinates. A 3×1 translation vector is used to compensate for the positional offset between the virtual scene and the physical scene; the multi-channel output unit includes a display terminal, audio equipment and haptic feedback device, used to output the fused exhibition content and interactive feedback information in multiple dimensions.
[0013] As a further aspect of the present invention: the multimodal perception module includes a visual acquisition unit, a motion capture unit, and a voice acquisition unit; the visual acquisition unit uses a combination of a depth camera and a high-definition camera to acquire user facial expression and gaze direction data; the motion capture unit uses an inertial motion capture device or an optical motion capture device to acquire user limb movements and posture data, and the smoothing and de-shaking processing of the posture data adopts a moving average algorithm. ;in, The pose data at time t is the smoothed pose data. The attitude data after smoothing at time t-1 The original attitude data was collected at time t. The smoothing coefficient, with a value range of [0.6, 0.9], is used to balance the weights of historical data and current data, and reduce noise interference during the action acquisition process; the voice acquisition unit uses an array microphone to acquire user voice commands and filter environmental noise.
[0014] As a further aspect of the present invention: the preprocessing process of the data processing module includes: image enhancement, denoising, and cropping of visual interaction data; smoothing, de-jittering, and coordinate standardization of action interaction data; and denoising, endpoint detection, and feature normalization of voice interaction data. The feature extraction process employs a deep learning model to extract visual features, action features, and voice features respectively, and obtains fused interaction features through a feature fusion algorithm. The feature fusion formula is as follows: ;in, To integrate interactive features, , , These are visual features, motion features, and voice features. , , These are the weight coefficients of each feature, and they satisfy... The semantic parsing process is based on a preset interaction intent dictionary to match and identify the fused interaction features and determine the user's interaction intent.
[0015] As a further aspect of the present invention, it also includes a network communication module, which employs a 5G or WiFi 6 communication protocol to enable data transmission between modules. The data transmission rate must meet the following requirements: ;in, This represents the actual data transmission rate. The amount of data required to be transmitted in a single interaction. An interaction delay threshold (not exceeding 100ms) is set to ensure the real-time nature of the interaction process; the network communication module can establish communication with a remote cloud platform to realize remote updates of virtual exhibition content and cloud backup of interactive data.
[0016] A multimodal perception method for a virtual-real integrated exhibition and display interactive system includes the following steps:
[0017] S1: Initialize system parameters, load preset virtual exhibition content and interaction rule library, and complete the initial alignment of the virtual-real fusion display module with the physical exhibition scene;
[0018] S2: Real-time collection of user visual interaction data, action interaction data, and voice interaction data through the multimodal perception module to form a multi-dimensional interactive data stream;
[0019] S3: Preprocess the multi-dimensional interactive data stream to eliminate noise and redundant data, and obtain standardized interactive data;
[0020] S4: Use a multimodal feature extraction model to extract features from standardized interaction data to obtain interaction features corresponding to each modality, and use an attention mechanism to weight and fuse the interaction features of each modality to obtain a fused feature vector;
[0021] S5: Input the fused feature vector into the intent recognition model, and combine it with the exhibition association data in the storage module to parse out the user's specific interaction intent;
[0022] S6: Generate corresponding control commands based on the specific interaction intent, drive the virtual-real fusion display module to adjust the display content and output interactive feedback.
[0023] As a further aspect of the present invention: in step S2, the acquisition frequency of the visual interaction data is not less than 30fps, the acquisition accuracy of the action interaction data is not less than 0.1°, and the sampling rate of the voice interaction data is not less than 16kHz; multimodal data uses a timestamp synchronization mechanism to achieve data alignment, and the timestamp synchronization error must meet the following requirements: ;in, , These represent the differences between the acquisition timestamps and the reference timestamps for any two modalities. To ensure that different modal data correspond to the same interaction time, the maximum allowable synchronization error is set (with a value not exceeding 5ms).
[0024] As a further aspect of the present invention: In step S4, the multimodal feature extraction model includes a visual feature extraction branch, an action feature extraction branch, and a speech feature extraction branch; the visual feature extraction branch uses a CNN model, the action feature extraction branch uses an LSTM model, and the speech feature extraction branch uses an MFCC+CNN model; the attention mechanism dynamically weights and fuses different modal features by calculating the information entropy weights of each modality feature, and the information entropy calculation formula is: ;in, Let F be the information entropy of a certain modal feature. For feature dimension, The probability of the k-th dimension of the feature is given. The larger the information entropy, the more effective information the modality feature contains, and the larger the corresponding weight coefficient.
[0025] As a further aspect of the present invention: In step S5, the intent recognition model adopts a Transformer-based classification model, and the model output is the probability distribution of each interaction intent: ;in, Given a fusion feature vector The conditional probability of each interaction intent y at time. This is the classification weight matrix. The bias vector is used to convert the model output into a value that conforms to the probability distribution; the preset interaction intents include content query intent, perspective switching intent, detail zooming intent, interactive experience intent, and exit interaction intent; the exhibition-related data includes exhibit background information, related exhibit recommendation data, and interaction history data.
[0026] As a further aspect of the present invention, it also includes step S7: recording user interaction process data, including interaction intent, interaction duration, and interaction feedback results; analyzing user preferences using data mining algorithms; and calculating user preference using the following formula: ;in, For users' preferences for the i-th type of exhibition content, Let i be the duration of user interaction with the i-th type of exhibition content. The total number of exhibition content categories, The interactive feedback score (with a value range of [0,1]) is used to quantify users' preferences for different categories of exhibition content, providing data support for the optimization of virtual exhibition content.
[0027] The beneficial effects of this invention are:
[0028] 1. More comprehensive interaction modalities and more accurate intent recognition: This invention simultaneously collects multi-dimensional interaction data such as user vision, action, and voice through a multi-modal perception module. Combined with the multi-modal feature fusion and semantic parsing technology of the data processing module, it can comprehensively capture user interaction behavior, significantly improve the accuracy of user interaction intent recognition, avoid intent misunderstanding caused by single-modal interaction, and optimize the user interaction experience.
[0029] 2. More precise integration of virtual and real elements, and a stronger sense of immersion: This invention uses a coordinate alignment algorithm in the virtual-real integration display module to achieve precise matching between virtual content and physical scenes, effectively solving the problem of misalignment between virtual and physical scenes in existing systems. Combined with the multi-dimensional display of multi-channel output units, it can create a more realistic immersive exhibition environment and enhance users' sense of participation and immersion.
[0030] 3. More real-time interactive response and higher system stability: This invention adopts a high-speed communication protocol through the network communication module and clearly defines the data transmission rate requirements, ensuring the real-time performance of multimodal data transmission; at the same time, it reduces data noise through preprocessing algorithms such as moving average, improving data quality and the stability of system interactive response, and reducing the occurrence of problems such as interaction delay and feedback misalignment.
[0031] 4. Enhanced personalized service capabilities and higher exhibition value: This invention records user interaction data and combines it with a user preference calculation model to quantify user preferences for different exhibition content. This provides precise data support for optimizing virtual exhibition content, enabling personalized exhibition content recommendations, improving the relevance and dissemination value of exhibitions. At the same time, cloud backup and update functions can expand the coverage and content flexibility of exhibitions. Attached Figure Description
[0032] Figure 1 This is a diagram illustrating the overall architecture of the virtual-real fusion exhibition system and the multimodal perception method of the present invention.
[0033] Figure 2 This is a flowchart illustrating the virtual-real fusion exhibition system and multimodal perception method of the present invention. Detailed Implementation
[0034] The present invention will be further described in detail below with reference to specific embodiments, so that those skilled in the art can more clearly understand the technical solution of the present invention and implement it. It should be noted that the following embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.
[0035] Example 1: Basic Virtual-Real Fusion Exhibition and Interactive System and Multimodal Perception Method
[0036] This embodiment is a basic configuration scheme, suitable for small and medium-sized offline exhibition scenarios (such as community exhibition halls and small museums). The core objective is to achieve a basic virtual-real integrated interactive experience while controlling costs.
[0037] 1. System Composition:
[0038] (1) Virtual-Real Fusion Display Module: The virtual content loading unit uses a regular PC processor (Intel i5-12400) to parse and load virtual exhibit models in FBX format; the scene fusion rendering unit uses monocular vision positioning technology to obtain spatial parameters of the physical scene, and the coordinate alignment algorithm adopts... The rotation matrix T and translation vector b are obtained through pre-calibration; the multi-channel output unit uses a 55-inch LCD screen, ordinary stereo sound, and a simple vibration haptic feedback handle.
[0039] (2) Multimodal perception module: The visual acquisition unit uses a 1080P high-definition camera to collect user facial expression and gaze direction data; the motion capture unit uses a low-cost inertial motion capture wristband to collect user hand posture data, and the posture data is smoothed and de-shaken using a moving average algorithm. The smoothing coefficient α is set to 0.7; the voice acquisition unit uses a single array microphone and filters environmental noise through a basic noise reduction algorithm.
[0040] (3) Other modules: The data processing module uses the same PC processor as the virtual content loading unit; the storage module uses a 1TB mechanical hard disk to store virtual exhibition content and exhibition-related data; the network communication module uses the WiFi 6 protocol to realize data transmission between modules and does not connect to the cloud platform.
[0041] 2. Implementation steps of multimodal sensing method:
[0042] Step S1: Initialize system parameters, load preset virtual exhibit models (such as historical artifact models) and interaction rule library (including two basic interaction intents: "view details" and "switch exhibits"), and complete the initial alignment of the virtual-real fusion display module with the physical exhibition hall through monocular visual positioning.
[0043] Step S2: Simultaneously collect user visual, hand movement, and voice data using a high-definition camera, inertial wristband, and single-array microphone. The visual data acquisition frequency is 30fps, the movement data acquisition accuracy is 0.5°, and the voice data sampling rate is 16kHz. Data alignment is achieved through a timestamp synchronization mechanism, and the synchronization error is controlled within 5ms.
[0044] Step S3: Preprocess the multi-dimensional interactive data stream, perform simple noise reduction and cropping on the visual data, smooth and de-jitter and standardize the coordinates on the motion data, and perform noise reduction and endpoint detection on the voice data to obtain standardized interactive data.
[0045] Step S4: Extract features for each modality using a lightweight multimodal feature extraction model (basic CNN + simple LSTM + MFCC), and then use the feature fusion algorithm as described in claim 4. Fusion features, where weight coefficients =0.3、 =0.4、 =0.3.
[0046] Step S5: Input the fused feature vector into the intent recognition model based on a simple neural network, and combine it with the exhibit background information in the storage module to parse the user's interaction intent.
[0047] Step S6: Generate control commands based on the interaction intent. For example, when the intent to "view details" is recognized, drive the display screen to zoom in on a part of the virtual exhibit; when the intent to "switch exhibits" is recognized, switch to display the next virtual exhibit.
[0048] Step S7: Record the user interaction duration and interaction intent, and calculate the preference using the formula described in claim 10. ( (Taking only 0 or 1 indicates whether a valid interaction was completed) Quantifies user preferences and provides basic data for subsequent adjustments to exhibition content.
[0049] Example 2: High-performance virtual-real fusion exhibition and interactive system and multimodal perception method
[0050] This embodiment is a high-performance configuration solution suitable for large-scale professional exhibition scenarios (such as provincial museums and science and technology exhibition halls). Its core objective is to achieve a high-precision, highly immersive virtual-real fusion interactive experience, supporting personalized services and remote sharing.
[0051] 1. System Composition:
[0052] (1) Virtual-Real Fusion Display Module: The virtual content loading unit uses a high-performance server processor (Intel Xeon E5-2697 v4) to parse and load a high-precision virtual exhibition model (including animation and 3D sound effects); the scene fusion rendering unit uses LiDAR + multi-view vision fusion positioning technology to obtain the spatial parameters of the physical scene, and the coordinate alignment algorithm adds a light and shadow compensation factor on the basis of claim 2, and the optimized coordinate alignment formula is as follows: (Where k is the light and shadow compensation coefficient, and L is the light and shadow intensity value of the actual scene); the multi-channel output unit adopts an 8K ultra-high-definition splicing screen, 5.1 channel surround sound, and high-precision force feedback gloves.
[0053] (2) Multimodal perception module: The visual acquisition unit adopts a combination of 4K depth camera and high-definition camera to collect user facial expression, gaze direction and body contour data; the motion capture unit adopts an optical motion capture system (including 8 capture cameras) to collect user full body posture data, and the smoothing coefficient α in the smoothing and shaking algorithm is 0.85; the voice acquisition unit adopts 8 array microphones, filters environmental noise through deep learning noise reduction algorithm, and supports far-field voice recognition.
[0054] (3) Other modules: The data processing module uses a GPU-accelerated server (NVIDIA Tesla V100) to improve the efficiency of feature extraction and intent recognition; the storage module uses a 2TB SSD + 10TB enterprise-grade hard disk array; the network communication module uses 5G + WiFi 6 dual protocols to realize high-speed data transmission between modules, and at the same time accesses the remote cloud platform to support remote updates of virtual exhibition content and cloud backup of interactive data.
[0055] 2. Implementation steps of multimodal sensing method:
[0056] Steps S1-S3: Initialize parameters to load high-precision virtual content and a rule base containing 5 types of interactive intents, and complete accurate alignment through LiDAR + multi-view vision; adopt higher frequency to collect multimodal data (visual 60fps, motion accuracy 0.1°, voice 48kHz), and control the synchronization error within 2ms; the preprocessing process adds steps such as image enhancement and feature normalization to improve data quality.
[0057] Step S4: Using the multimodal feature extraction model (CNN+LSTM+MFCC+CNN) described in claim 8, features are fused through an attention mechanism, with attention weights calculated based on information entropy. Dynamic allocation.
[0058] Step S5: Employ a Transformer-based intent recognition model to output the probability distribution as described in claim 9. By combining exhibition-related data synchronized in the cloud with historical user interaction data, user intent can be accurately identified.
[0059] Step S6: Generate multi-dimensional feedback instructions based on the intent. For example, when the intent of "interactive experience" is recognized, drive the force feedback glove to simulate the tactile sensation of touching the exhibit, and output the corresponding sound effect through the surround sound.
[0060] Step S7: Record all interaction data, quantify user preferences through a precise preference calculation model, and synchronize the results to the cloud to provide data support for personalized exhibition content recommendations and exhibition sharing across the country.
[0061] Example 3: Lightweight Virtual-Real Fusion Exhibition and Interactive System and Multimodal Perception Method
[0062] This embodiment is a lightweight configuration solution suitable for mobile exhibition scenarios (such as mobile exhibition halls and small-scale online and offline integrated exhibitions). The core objective is to achieve system miniaturization and portability, and support rapid deployment.
[0063] 1. System Composition:
[0064] (1) Virtual-Real Fusion Display Module: The virtual content loading unit uses an embedded processor (NVIDIA Jetson Nano) to load lightweight virtual exhibition content (such as simplified 2D to 3D models); the scene fusion rendering unit uses a mobile phone camera to achieve simple spatial positioning, and the coordinate alignment algorithm is simplified to two-dimensional coordinate alignment. (T is a 2×2 rotation matrix, b is a 2×1 translation vector); the multi-channel output unit uses a tablet computer display and Bluetooth headset.
[0065] (2) Multimodal perception module: The visual acquisition unit reuses the mobile phone camera; the motion capture unit uses a tablet computer touch screen + gyroscope to collect user touch actions and device posture data; the voice acquisition unit reuses the tablet computer's built-in microphone and processes noise through the system's built-in noise reduction function.
[0066] (3) Other modules: The data processing module and the virtual content loading unit share the embedded processor; the storage module uses a 32GB SD card; the network communication module uses the WiFi 6 protocol and supports access to a small local server to achieve fast content updates.
[0067] 2. Implementation steps of multimodal sensing method:
[0068] The steps described in claim 6 are simplified, with a focus on optimizing the lightweight nature of data acquisition and processing. This includes reducing the data acquisition frequency (20fps for visual processing), adopting a minimalist feature extraction model, and reducing the number of interaction intent categories (retaining only "View" and "Exit") to ensure efficient operation of the system on embedded hardware, enabling rapid deployment and use.
[0069] III. Comparative Analysis of Examples
[0070] To clearly illustrate the differences and applicable scenarios of each embodiment, the above three embodiments are compared from the following six dimensions, as follows:
[0071] 1. Configuration Cost Dimension: Example 1 (Basic Type) has the lowest cost, with the total cost of core hardware (PC, ordinary display device, low-cost sensor) approximately 15,000 yuan; Example 2 (High-Performance Type) has the highest cost, with the total cost of core hardware such as server, LiDAR, and optical motion capture system approximately 500,000 yuan; Example 3 (Lightweight Type) has a lower cost, with the total cost of core hardware such as embedded processor and tablet computer approximately 8,000 yuan. The cost differences mainly stem from the differences in the configuration of positioning technology, sensing device accuracy, and data processing capabilities.
[0072] 2. Interactive Experience Dimension: Example 2 offers the best interactive experience. High-precision optical motion capture, 8K display, and force feedback devices enable full-dimensional, highly immersive interaction, with an intent recognition accuracy of over 95%. Example 1 provides a basic interactive experience with an intent recognition accuracy of approximately 80%, meeting the basic needs of small and medium-sized exhibitions. Example 3 is limited by hardware and can only achieve simple touch + voice interaction with an intent recognition accuracy of approximately 75%, but it excels in ease of operation and lacks a learning curve.
[0073] 3. Applicable Scenarios: Example 1 is suitable for small and medium-sized fixed exhibition scenarios such as community exhibition halls and small museums, balancing cost and basic interaction needs; Example 2 is suitable for large-scale professional exhibition scenarios such as provincial museums and science and technology exhibition halls, pursuing high-end immersive experiences and personalized services; Example 3 is suitable for mobile scenarios such as mobile exhibition halls, campus tours, and small online and offline integrated exhibitions, with the core advantages being portability and rapid deployment.
[0074] 4. Data processing capability dimension: Example 2 uses a GPU-accelerated server, which can process high-precision, high-frequency multimodal data, supports complex feature fusion and intent recognition algorithms, and has a data processing latency of less than 30ms; Example 1 uses a regular PC processor, which can only process basic multimodal data, with a data processing latency of about 50-80ms; Example 3 uses an embedded processor, which can only process simplified multimodal data, with a data processing latency of about 80-100ms, but can meet the real-time requirements of lightweight interaction.
[0075] 5. Scalability dimension: Example 2 supports 5G+cloud platform access, enabling remote updates of virtual exhibition content, cloud backup of interactive data, and nationwide exhibition sharing, with the strongest scalability; Example 1 only supports local content updates without cloud interaction, with weaker scalability; Example 3 supports access to a small local server to achieve rapid content updates, which can meet the flexible adjustment needs of mobile exhibitions, with moderate scalability.
[0076] 6. Deployment difficulty dimension: Example 3 has the lowest deployment difficulty, requiring only a simple connection between the embedded processor and the tablet computer, and deployment can be completed within 30 minutes; Example 1 has a medium deployment difficulty, requiring the connection between the PC and various sensing devices and spatial positioning calibration, and deployment can be completed in about 2-3 hours; Example 2 has the highest deployment difficulty, requiring professional personnel to complete the installation and debugging of LiDAR and optical capture cameras, and multi-device collaborative calibration, and deployment can be completed in about 1-2 days.
[0077] In summary, the three embodiments are designed for different cost budgets and scenario requirements, forming a complete solution covering basic, high-end, and lightweight aspects. The appropriate implementation method can be flexibly selected according to the actual exhibition scale, capital investment, and usage scenario, fully demonstrating the flexibility and practicality of the present invention.
[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A virtual-real integrated exhibition and display interactive system, characterized in that, include: The system includes a virtual-real fusion display module, a multimodal perception module, an interactive control module, a data processing module, and a storage module. The virtual-real fusion display module is used to load preset virtual exhibition content and output the virtual exhibition content and the physical exhibition scene in real time after fusion rendering. The multimodal perception module is used to collect multi-dimensional interaction data of users, which includes at least visual interaction data, action interaction data and voice interaction data. The data processing module is connected to the multimodal perception module and the storage module respectively, and is used to preprocess, extract features and perform semantic parsing on the multi-dimensional interactive data to obtain the user's interaction intent, and call the exhibition-related data pre-stored in the storage module. The interactive control module is connected to the data processing module and the virtual-real fusion display module respectively. It is used to generate corresponding display adjustment instructions and interactive feedback instructions according to the user's interaction intention and exhibition-related data, and send them to the virtual-real fusion display module. The virtual-real fusion display module responds to the display adjustment command and interactive feedback command, adjusts the display status of the virtual exhibition content, and outputs corresponding interactive feedback information.
2. The virtual-real integrated exhibition and display interactive system according to claim 1, characterized in that, The virtual-real fusion display module includes a virtual content loading unit, a scene fusion rendering unit, and a multi-channel output unit. The virtual content loading unit parses and loads virtual exhibition models, animations, and audio resources in a preset format. The scene fusion rendering unit obtains spatial parameters of the physical exhibition scene using spatial positioning technology and employs a coordinate alignment algorithm to achieve precise matching between the virtual exhibition content and the physical exhibition scene. The coordinate alignment formula is: ;in, The three-dimensional coordinates representing the content of the virtual exhibition. Representing the three-dimensional coordinates of a physical exhibition scene. It is a 3×3 rotation matrix used to align the virtual coordinates with the physical coordinates. A 3×1 translation vector is used to compensate for the positional offset between the virtual scene and the physical scene; the multi-channel output unit includes a display terminal, audio equipment and haptic feedback device, used to output the fused exhibition content and interactive feedback information in multiple dimensions.
3. The virtual-real integrated exhibition and display interactive system according to claim 1, characterized in that, The multimodal perception module includes a visual acquisition unit, a motion capture unit, and a voice acquisition unit. The visual acquisition unit uses a combination of a depth camera and a high-definition camera to acquire user facial expression and gaze direction data. The motion capture unit uses inertial motion capture equipment or optical motion capture equipment to acquire user limb movements and posture data. The posture data smoothing and de-shaking processing uses a moving average algorithm. ;in, The pose data at time t is the smoothed pose data. The attitude data after smoothing at time t-1 The original attitude data was collected at time t. The smoothing coefficient, with a value range of [0.6, 0.9], is used to balance the weights of historical data and current data, and reduce noise interference during the action acquisition process; the voice acquisition unit uses an array microphone to acquire user voice commands and filter environmental noise.
4. The virtual-real integrated exhibition and display interactive system according to claim 1, characterized in that, The preprocessing process of the data processing module includes: image enhancement, denoising, and cropping of visual interaction data; smoothing, de-jittering, and coordinate standardization of action interaction data; and denoising, endpoint detection, and feature normalization of voice interaction data. The feature extraction process uses a deep learning model to extract visual features, action features, and voice features respectively, and obtains fused interaction features through a feature fusion algorithm. The feature fusion formula is as follows: ;in, To integrate interactive features, , , These are visual features, motion features, and voice features. , , These are the weight coefficients of each feature, and they satisfy... The semantic parsing process is based on a preset interaction intent dictionary to match and identify the fused interaction features and determine the user's interaction intent.
5. The virtual-real integrated exhibition and display interactive system according to claim 1, characterized in that, It also includes a network communication module, which uses 5G or WiFi 6 communication protocols to enable data transmission between modules. The data transmission rate must meet the following requirements: ;in, This represents the actual data transmission rate. The amount of data required to be transmitted in a single interaction. An interaction delay threshold (not exceeding 100ms) is set to ensure the real-time nature of the interaction process; the network communication module can establish communication with a remote cloud platform to realize remote updates of virtual exhibition content and cloud backup of interactive data.
6. A multimodal perception method for a virtual-real fusion exhibition and interactive system according to any one of claims 1-5, characterized in that, Includes the following steps: S1: Initialize system parameters, load preset virtual exhibition content and interaction rule library, and complete the initial alignment of the virtual-real fusion display module with the physical exhibition scene; S2: Real-time collection of user visual interaction data, action interaction data, and voice interaction data through the multimodal perception module to form a multi-dimensional interactive data stream; S3: Preprocess the multi-dimensional interactive data stream to eliminate noise and redundant data, and obtain standardized interactive data; S4: Use a multimodal feature extraction model to extract features from standardized interaction data to obtain interaction features corresponding to each modality, and use an attention mechanism to weight and fuse the interaction features of each modality to obtain a fused feature vector; S5: Input the fused feature vector into the intent recognition model, and combine it with the exhibition association data in the storage module to parse out the user's specific interaction intent; S6: Generate corresponding control commands based on the specific interaction intent, drive the virtual-real fusion display module to adjust the display content and output interactive feedback.
7. The multimodal perception method for a virtual-real integrated exhibition and display interactive system according to claim 6, characterized in that, In step S2, the acquisition frequency of the visual interaction data is no less than 30fps, the acquisition accuracy of the action interaction data is no less than 0.1°, and the sampling rate of the voice interaction data is no less than 16kHz; multimodal data uses a timestamp synchronization mechanism to achieve data alignment, and the timestamp synchronization error must meet the following requirements: ;in, , These represent the differences between the acquisition timestamps and the reference timestamps for any two modalities. To ensure that different modal data correspond to the same interaction time, the maximum allowable synchronization error is set (with a value not exceeding 5ms).
8. The multimodal perception method for a virtual-real integrated exhibition and display interactive system according to claim 6, characterized in that, In step S4, the multimodal feature extraction model includes a visual feature extraction branch, an action feature extraction branch, and a speech feature extraction branch; The visual feature extraction branch uses a CNN model, the action feature extraction branch uses an LSTM model, and the speech feature extraction branch uses an MFCC+CNN model. The attention mechanism dynamically weights and fuses different modal features by calculating the information entropy weights of each modality feature. The information entropy calculation formula is as follows: ;in, Let F be the information entropy of a certain modal feature. For feature dimension, The probability of the k-th dimension of the feature is given. The larger the information entropy, the more effective information the modality feature contains, and the larger the corresponding weight coefficient.
9. The multimodal perception method for a virtual-real integrated exhibition and display interactive system according to claim 6, characterized in that, In step S5, the intent recognition model adopts a Transformer-based classification model, and the model output is the probability distribution of each interaction intent: ;in, Given a fused feature vector The conditional probability of each interaction intent y at time. This is the classification weight matrix. The bias vector is used to convert the model output into a value that conforms to the probability distribution; the preset interaction intents include content query intent, perspective switching intent, detail zooming intent, interactive experience intent, and exit interaction intent; the exhibition-related data includes exhibit background information, related exhibit recommendation data, and interaction history data.
10. The multimodal perception method for a virtual-real integrated exhibition and display interactive system according to claim 6, characterized in that, It also includes step S7: recording user interaction process data, including interaction intent, interaction duration, and interaction feedback results, and analyzing user preferences through data mining algorithms. The user preference calculation formula is: ;in, For users' preference for the i-th type of exhibition content, Let i be the duration of user interaction with the i-th type of exhibition content. The total number of exhibition content categories, The interactive feedback score (with a value range of [0,1]) is used to quantify users' preferences for different categories of exhibition content, providing data support for the optimization of virtual exhibition content.