Virtual immersive interactive display system

Abstract

The invention relates to the technical field of virtual reality, in particular to a virtual immersive interactive display system which comprises a multi-modal sensing module, a data preprocessing module, a data analysis module, a content generation module, an efficient rendering module, a personalized recommendation unit, a display module, a data fusion module, a feedback module and a physiological status monitoring module. The data preprocessing module is connected with the multi-modal sensing module, the data analysis module is connected with the data preprocessing module, the data fusion module is connected with the data analysis module, the personalized recommendation unit and the physiological state detection unit are also connected with the data fusion module, and the content generation module is connected with the data fusion module. And the efficient rendering module is connected with the content generation module, so that the technical problems that in the prior art, a virtual immersive interactive display system lacks physiological state monitoring, historical data acquisition and dynamic adjustment functions, so that content generation is single, and diversified requirements of users cannot be met are solved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a virtual immersive interactive display system. Background Art

[0002] With the rapid development of science and technology, virtual reality (VR) technology has become a hot topic in today's society, demonstrating enormous potential for application in numerous fields, including education, training, and entertainment. Virtual immersive interactive display systems, as cutting-edge applications of VR technology, simulate real or imaginary environments, allowing users to immerse themselves in virtual worlds and interact naturally and intuitively. This novel display method not only greatly enriches the means of information transmission but also significantly enhances the user experience.

[0003] However, although the existing virtual immersive interactive display systems have achieved certain achievements in providing immersive experience, they lack physiological status monitoring, historical data acquisition and dynamic adjustment functions, resulting in single content generation and inability to meet the diverse needs of users. Summary of the Invention

[0004] The purpose of the present invention is to provide a virtual immersive interactive display system, which aims to solve the technical problem that the virtual immersive interactive display system in the existing technology lacks physiological status monitoring, historical data acquisition and dynamic adjustment functions, resulting in single content generation and inability to meet the diverse needs of users.

[0005] To achieve the above objectives, the present invention adopts a virtual immersive interactive display system, which includes a multimodal perception module, a data preprocessing module, a data parsing module, a content generation module, an efficient rendering module, a personalized recommendation unit, a display module, a data fusion module, a feedback module and a physiological status monitoring module;

[0006] The data preprocessing module is connected to the multimodal perception module, the data parsing module is connected to the data preprocessing module, the data fusion module is connected to the data parsing module, the personalized recommendation unit and the physiological state detection unit are also connected to the data fusion module, the content generation module is connected to the data fusion module, the efficient rendering module is connected to the content generation module, the display module is connected to the efficient rendering module, and the feedback module is connected to the display module;

[0007] The multimodal perception module is used to collect on-site data, including environmental data and user action data, and transmit it to the data preprocessing module to preprocess the collected multimodal data;

[0008] The physiological status monitoring module monitors the user's physiological status data in real time and transmits the data to the data fusion module;

[0009] The data analysis module analyzes the pre-processed data based on the deep learning model to obtain key information;

[0010] The data fusion module integrates the parsed data with personalized recommendation data and physiological status data to form a comprehensive user profile;

[0011] The content generation module receives the user portrait transmitted by the data fusion module and uses generative adversarial networks (GANs) and reinforcement learning to dynamically generate display content based on user behavior and environmental changes;

[0012] The efficient rendering module adopts advanced graphics rendering technology to efficiently render the generated content and then display it through the display module.

[0013] The personalized recommendation unit includes a historical data acquisition module, a feedback receiving module and a data mining module. The data fusion module is connected to the content generation module, and the data mining module is connected to the historical data acquisition module and the feedback receiving module.

[0014] The virtual immersive interactive display system further includes a facial emotion perception module and an emotion data generation module, and the emotion data generation module is connected to the facial emotion perception module and the data fusion module;

[0015] The facial emotion perception module uses Dlib or MediaPipe Face Mesh to extract facial feature points and achieves high-precision emotion recognition based on pre-trained convolutional neural networks (such as ResNet-50+LSTM) combined with micro-expression analysis technology;

[0016] The emotion data generation module captures the changing trend of emotions over time through LSTM or Transformer and transmits it to the data fusion module.

[0017] The virtual immersive interactive display system further includes a music soothing module and a voice soothing module, and both the music soothing module and the voice soothing module are connected to the data fusion module.

[0018] The virtual immersive interactive display system further includes a security monitoring module and an alarm module, and the security monitoring module is connected to the multimodal perception module and the alarm module.

[0019] Wherein, the virtual immersive interactive display system further includes an environment simulation module, and the environment simulation module is connected to the efficient rendering module;

[0020] The environmental simulation module is used to simulate different environmental conditions.

[0021] Wherein, the virtual immersive interactive display system further includes a content management module, and the content management module is connected to the content generation module;

[0022] The content management module is used to manage the content library generated by the content generation module.

[0023] Wherein, the virtual immersive interactive display system further includes a security authentication module, and the security authentication module is connected to the multimodal perception module;

[0024] The security authentication module allows users to perform identity authentication and data encryption to ensure the security and privacy protection of the system.

[0025] A virtual immersive interactive display system of the present invention, during specific use, first, the multimodal perception module collects on-site data, including environmental data and user action data, and transmits it to the data preprocessing module for preprocessing the collected multimodal data; the physiological state monitoring module monitors the user's physiological state data in real time and transmits it to the data fusion module; the data parsing module analyzes the preprocessed data based on a deep learning model to obtain key information; the data fusion module fuses the parsed data with personalized recommendation data and physiological state data to form a comprehensive user profile; the content generation module receives the user profile received from the data fusion module, and uses generative adversarial networks (GANs) and reinforcement learning to dynamically generate display content based on user behavior and environmental changes; the efficient rendering module uses advanced graphics rendering technology to efficiently render the generated content and then display it through the display module. This solves the technical problem that virtual immersive interactive display systems in the prior art lack physiological state monitoring, historical data acquisition, and dynamic adjustment functions, resulting in single content generation and an inability to meet the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0028] Figure 2It is a principle block diagram of the second embodiment of the present invention.

[0029] Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0030] 101-Multimodal perception module, 102-Data preprocessing module, 103-Data analysis module, 104-Content generation module, 105-Efficient rendering module, 106-Personalized recommendation unit, 107-Display module, 108-Data fusion module, 109-Feedback module, 110-Physiological state monitoring module, 111-Facial emotion perception module, 112-Emotional data generation module, 113-Music soothing module, 114-Voice soothing module, 115-Historical data acquisition module, 116-Feedback receiving module, 117-Data mining module, 201-Security monitoring module, 202-Alarm module, 203-Environmental simulation module, 204-Content management module, 205-Security authentication module, 301-Login module, 302-Authentication module, 303-Permission management module. DETAILED DESCRIPTION

[0031] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0032] The first embodiment of this application is:

[0033] See also Figure 1 , Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0034] The present invention provides a virtual immersive interactive display system, including a multimodal perception module 101, a data preprocessing module 102, a data analysis module 103, a content generation module 104, an efficient rendering module 105, a personalized recommendation unit 106, a display module 107, a data fusion module 108, a feedback module 109, a physiological state monitoring module 110, a facial emotion perception module 111, an emotion data generation module 112, a music soothing module 113 and a voice soothing module 114; the personalized recommendation unit 106 includes a historical data acquisition module 115, a feedback receiving module 116 and a data mining module 117. The above solution solves the technical problem that the virtual immersive interactive display system in the prior art lacks physiological state monitoring, historical data acquisition and dynamic adjustment functions, resulting in single content generation and an inability to meet the diverse needs of users.

[0035] For this specific embodiment, the multimodal perception module 101 adopts a multi-source information fusion algorithm to collect field data. The algorithm can simultaneously process environmental data (such as light intensity, sound frequency, etc.) and user action data (such as body movements, facial expressions, etc.), and accurately transmit these multimodal data to the data preprocessing module 102.

[0036] In the data preprocessing module 102 , data cleaning and feature extraction algorithms are used to preprocess the collected multimodal data to eliminate noise and extract features useful for subsequent analysis.

[0037] The physiological status monitoring module 110 uses a biological signal monitoring algorithm to monitor the user's physiological status data in real time, such as heart rate variability, blood pressure fluctuations, etc. These data are instantly transmitted to the data fusion module 108 through an efficient data transmission protocol.

[0038] The data analysis module 103 performs an in-depth analysis on the pre-processed data based on an analysis algorithm combining a convolutional neural network (CNN) and a recurrent neural network (RNN) in deep learning, thereby extracting key information such as user behavior patterns and interest preferences.

[0039] The data fusion module 108 uses advanced data integration and mining algorithms to organically integrate the parsed data with the data generated by the personalized recommendation algorithm (such as recommended content based on user historical behavior) and physiological status data, thereby constructing a comprehensive and accurate user portrait.

[0040] After receiving the user portrait from the data fusion module 108, the content generation module 104 uses the creative generation capability of generative adversarial networks (GANs) and the decision optimization mechanism of reinforcement learning to dynamically generate personalized and attractive display content based on the user's current behavior pattern, environmental changes, and possible emotional state.

[0041] Finally, the efficient rendering module 105 uses advanced graphics rendering algorithms, such as ray tracing and global illumination, to perform precise and efficient rendering of the generated content, ensuring a realistic and immersive display. The rendered content is then presented to the user in an intuitive and vivid manner via the display module 107.

[0042] Among them, the data preprocessing module 102 is connected to the multimodal perception module 101, the data analysis module 103 is connected to the data preprocessing module 102, the data fusion module 108 is connected to the data analysis module 103, the personalized recommendation unit 106 and the physiological state detection unit are also connected to the data fusion module 108, the content generation module 104 is connected to the data fusion module 108, the efficient rendering module 105 is connected to the content generation module 104, the display module 107 is connected to the efficient rendering module 105, and the feedback module 109 is connected to the display module 107. When used specifically, the multimodal perception module 101 first collects on-site data, including environmental data and user action data, and transmits it to the data preprocessing module 102 to preprocess the collected multimodal data; the physiological state monitoring module 110 implements The user's physiological state data is monitored in real time and transmitted to the data fusion module 108; the data analysis module 103 analyzes the pre-processed data based on the deep learning model to obtain key information; the data fusion module fuses the analyzed data with the personalized recommendation data and the physiological state data to form a comprehensive user portrait; the content generation module 104 receives the user portrait received from the data fusion module 108, and uses generative adversarial networks (GANs) and reinforcement learning to dynamically generate display content according to user behavior and environmental changes; the efficient rendering module 105 uses advanced graphics rendering technology to efficiently render the generated content and then display it through the display module 107, thereby solving the technical problem that the virtual immersive interactive display system in the existing technology lacks physiological state monitoring, historical data acquisition and dynamic adjustment functions, resulting in single content generation and inability to meet the diverse needs of users.

[0043] Secondly, the historical data acquisition module 115, the feedback receiving module 116 and the data mining module 117, the data fusion module 108 is connected to the content generation module 104, and the data mining module 117 is connected to the historical data acquisition module 115 and the feedback receiving module 116. The historical data acquisition module 115 adopts a big data storage and retrieval algorithm to efficiently collect and store the user's long-term interaction data. The feedback receiving module 116 uses a real-time communication protocol to obtain the user's real-time feedback on the displayed content. The data mining module 117 adopts a machine learning algorithm (such as association rule mining, cluster analysis, etc.) to deeply mine historical data and real-time feedback, discover the user's potential interest preferences and behavior patterns, and transmit them to the data fusion module 108.

[0044] At the same time, the emotion data generation module 112 is connected to the facial emotion perception module 111 and the data fusion module 108;

[0045] The facial emotion perception module 111 uses Dlib or MediaPipe Face Mesh to extract facial feature points and achieves high-precision emotion recognition based on a pre-trained convolutional neural network (such as ResNet-50+LSTM) combined with micro-expression analysis technology;

[0046] The emotion data generation module 112 captures the changing trend of emotions over time through LSTM or Transformer and transmits the captured data to the data fusion module 108 .

[0047] In addition, the music soothing module 113 and the voice soothing module 114 are both connected to the data fusion module 108. The music soothing module 113 adopts an intelligent music recommendation algorithm to select appropriate music types and rhythms according to the user's emotional state and environmental atmosphere to create a relaxing and pleasant atmosphere.

[0048] The voice soothing module 114 uses speech synthesis and emotion expression algorithms to generate appealing voice content, such as guiding words and storytelling, to help users relieve stress and relax.

[0049] When using a virtual immersive interactive display system of this embodiment, first, the multimodal perception module 101 collects field data, including environmental data and user action data, and transmits it to the data preprocessing module 102 for preprocessing the collected multimodal data; the physiological state monitoring module 110 monitors the user's physiological state data in real time and transmits it to the data fusion module 108; the data parsing module 103 analyzes the preprocessed data based on the deep learning model to obtain key information; the data fusion module combines the parsed data with the personalized recommendation data and the physiological state data. The data fusion module 108 receives the user portrait received by the user fusion module 108, and uses generative adversarial networks (GANs) and reinforcement learning to dynamically generate display content according to user behavior and environmental changes. The efficient rendering module 105 uses advanced graphics rendering technology to efficiently render the generated content and then display it through the display module 107. In this way, the technical problem that the virtual immersive interactive display system in the existing technology lacks physiological status monitoring, historical data acquisition and dynamic adjustment functions, resulting in single content generation and inability to meet the diverse needs of users is solved.

[0050] The second embodiment of the present application is:

[0051] Based on the first embodiment, please refer to Figure 2 , Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0052] The present invention provides a virtual immersive interactive display system, which also includes a security monitoring module 201, an alarm module 202, an environment simulation module 203, a content management module 204 and a security authentication module 205.

[0053] For this specific embodiment, the security monitoring module 201 is connected to the multimodal perception module 101 and the alarm module 202. The security monitoring module 201 monitors the abnormal behavior or status of the user and the environment in real time. Once a potential danger or abnormal situation is detected, an alarm is immediately issued through the alarm module 202 to ensure the safety of the user in the virtual immersive environment and prevent accidents.

[0054] The environment simulation module 203 is connected to the efficient rendering module 105 . The environment simulation module 203 utilizes a physical engine and an environment modeling algorithm to simulate realistic weather, lighting, sound and other environmental conditions.

[0055] Secondly, the content management module 204 is connected to the content generation module 104 . The content management module 204 uses a content management system (CMS) algorithm to classify, store, update, and distribute the content library generated by the content generation module 104 .

[0056] Again, the security authentication module 205 is connected to the multimodal sensing module 101. The security authentication module 205 adopts biometric recognition algorithm and encryption technology to obtain the user's biometric information, such as fingerprint, facial recognition, etc., through the multimodal sensing module 101 to perform identity authentication and encrypt the transmitted data.

[0057] When using a virtual immersive interactive display system of this embodiment, the security monitoring module 201 monitors abnormal behaviors or states of users and the environment in real time. Once potential dangers or abnormal situations are detected, an alarm is immediately issued through the alarm module 202 to ensure the safety of users in the virtual immersive environment and prevent accidents. The content management module 204 uses a content management system (CMS) algorithm to classify, store, update, and distribute the content library generated by the content generation module 104.

[0058] The third embodiment of the present application is:

[0059] Based on the second embodiment, please refer to Figure 3 , Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0060] The present invention provides a virtual immersive interactive display system, which also includes a login module 301 , an identity authentication module 302 and a rights management module 303 .

[0061] In this specific embodiment, the login module 301 is connected to the content management module 204, and the identity verification module 302 is connected to the login module 301. The login module 301 is used by administrators to log in to the content management module 204 to manage content resources. The identity verification module 302 strictly verifies the identity information provided by the user when logging in. The login module 301 will only allow the user to enter the system if the identity information entered by the user completely matches the system's preset information.

[0062] The authority management module 303 is connected to the identity authentication module 302 , and performs authority management according to the identity of the administrator who logs into the identity authentication module 302 to avoid overstepping of access and operation.

[0063] In the virtual immersive interactive display system of this embodiment, the login module 301 is used by administrators to log in to the content management module 204 to manage content resources. The identity authentication module 302 strictly verifies the identity information provided by the user during login. Only when the identity information entered by the user fully matches the system's preset information will the login module 301 allow the user to enter the system. The permission management module 303 manages permissions based on the identity of the administrator who logged into the identity authentication module 302, preventing unauthorized access and manipulation.

[0064] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A virtual immersive interactive display system, characterized in that: It includes multimodal perception module, data preprocessing module, data parsing module, content generation module, efficient rendering module, personalized recommendation unit, display module, data fusion module, feedback module and physiological status monitoring module; The data preprocessing module is connected to the multimodal perception module, the data parsing module is connected to the data preprocessing module, the data fusion module is connected to the data parsing module, the personalized recommendation unit and the physiological state detection unit are also connected to the data fusion module, the content generation module is connected to the data fusion module, the efficient rendering module is connected to the content generation module, the display module is connected to the efficient rendering module, and the feedback module is connected to the display module; The multimodal perception module is used to collect on-site data, including environmental data and user action data, and transmit it to the data preprocessing module to preprocess the collected multimodal data; The physiological status monitoring module monitors the user's physiological status data in real time and transmits the data to the data fusion module; The data analysis module analyzes the pre-processed data based on the deep learning model to obtain key information; The data fusion module integrates the parsed data with personalized recommendation data and physiological status data to form a comprehensive user profile; The content generation module receives the user portrait transmitted by the data fusion module and uses generative adversarial networks and reinforcement learning to dynamically generate display content based on user behavior and environmental changes; The efficient rendering module adopts advanced graphics rendering technology to efficiently render the generated content and then display it through the display module.

2. The virtual immersive interactive display system according to claim 1, characterized in that: The personalized recommendation unit includes a historical data acquisition module, a feedback receiving module and a data mining module. The data fusion module is connected to the content generation module, and the data mining module is connected to the historical data acquisition module and the feedback receiving module.

3. The virtual immersive interactive display system according to claim 2, characterized in that: The virtual immersive interactive display system further includes a facial emotion perception module and an emotion data generation module, and the emotion data generation module is connected to the facial emotion perception module and the data fusion module; The facial emotion perception module uses Dlib or MediaPipe Face Mesh to extract facial feature points and achieves high-precision emotion recognition based on pre-trained convolutional neural networks and micro-expression analysis technology; The emotion data generation module captures the changing trend of emotions over time through LSTM or Transformer and transmits it to the data fusion module.

4. The virtual immersive interactive display system according to claim 3, characterized in that: The virtual immersive interactive display system also includes a music soothing module and a voice soothing module, and both the music soothing module and the voice soothing module are connected to the data fusion module.

5. The virtual immersive interactive display system according to claim 4, characterized in that: The virtual immersive interactive display system also includes a security monitoring module and an alarm module, and the security monitoring module is connected to the multimodal perception module and the alarm module.

6. The virtual immersive interactive display system according to claim 5, characterized in that: The virtual immersive interactive display system further includes an environment simulation module, which is connected to the efficient rendering module; The environmental simulation module is used to simulate different environmental conditions.

7. The virtual immersive interactive display system according to claim 6, characterized in that: The virtual immersive interactive display system further includes a content management module, which is connected to the content generation module; The content management module is used to manage the content library generated by the content generation module.

8. The virtual immersive interactive display system according to claim 7, characterized in that: The virtual immersive interactive display system further includes a security authentication module, which is connected to the multimodal perception module; The security authentication module allows users to perform identity authentication and data encryption to ensure the security and privacy protection of the system.

Images