Virtual Reality-Based Thematic Display Method, System, and Electronic Device

Abstract

The present application provides a themed display method, system and electronic device based on virtual reality, which relates to the technical field of themed display. The method includes: connecting to a VR system to load a theme configuration file, and identifying a first set of display features of a first display theme; connecting to a modality mapping engine, performing a modality requirement analysis on the first set of display features, and determining a first effective modality combination corresponding to the first display theme; controlling a first group of perception devices to collect interaction events in the virtual reality environment of the VR system by using the first group of perception devices, and outputting a first effective modality data set; generating corresponding three-dimensional display content and displaying it in the virtual reality environment. By means of the present application, the technical problem that it is difficult to adapt to the differences in display themes due to the fixed virtual reality modality interaction method is solved. The data types required for important features related to the display theme are obtained through multi-modal perception, and the device is driven to perform virtual reality manifestation, thereby improving the adaptability of themed display.

Description

Technical Field

[0001] This application relates to the technical field of theme display, and particularly to a themed display method, system, and electronic device based on virtual reality. Background Art

[0002] Mainstream VR interactions rely on limited input methods (such as controller buttons, gesture recognition, voice commands). Their interaction logic and feedback modes are mostly fixed templates programmed in advance. Usually, content is selected according to a preset process or interface buttons, resulting in a lack of intelligent recognition and dynamic recommendation in the display, leading to a low matching degree with the theme content and a fragmented experience. Secondly, the modal interaction method of a virtual reality (VR) system is usually fixed, meaning that regardless of how the display theme changes, the user's interaction method with the virtual environment remains the same, resulting in the inability to provide the best user experience according to different display themes because different themes may require different interaction methods to maximize their immersion and interactivity.

[0003] In summary, there is a technical problem in the prior art that due to the fixed modal interaction method of virtual reality, it is difficult to adapt to the differentiated requirements of different display themes. Summary of the Invention

[0004] The purpose of this application is to provide a themed display method, system, and electronic device based on virtual reality to solve the technical problem in the prior art that due to the fixed modal interaction method of virtual reality, it is difficult to adapt to the differentiated requirements of different display themes.

[0005] In view of the above problems, this application provides a themed display method, system, and electronic device based on virtual reality.

[0006] In a first aspect, this application provides a themed display method based on virtual reality. The themed display method based on virtual reality is implemented through a themed display system based on virtual reality. Among them, the themed display method based on virtual reality includes: connecting to the VR system to load a theme configuration file, and identifying a first display feature set of a first display theme in the theme configuration file; connecting to a modal mapping engine, and performing modal requirement analysis on the first display feature set according to the modal mapping engine to determine a first effective modal combination corresponding to the first display theme; the VR system controls the loading of a corresponding first perception device group from all perception devices according to the first effective modal combination, collects interaction events in the virtual reality environment of the VR system by using the first perception device group, and outputs a first effective modal data set corresponding to the first effective modal combination; generating three-dimensional display content corresponding to the first effective modal data set, and displaying the three-dimensional display content in the virtual reality environment.

[0007] Optionally, identify a first set of display features of the first display theme in the theme profile, where each field of the theme profile includes a theme number, a theme type, and display content; read the fields of the theme profile, and parse the key display features of the first display theme based on NLP technology, and output them as the first set of display features. The key display features include physical entities, action elements, and interaction nodes.

[0008] Optionally, multiple modality components are encapsulated in the modality mapping engine, and the multiple modality components are connected to multiple sensing devices. Among them, the multiple modality components at least include a visual guidance modality, a force feedback modality, an action capture modality, an audio modality, and a spatial navigation modality; the modality mapping engine maps and analyzes the required modalities corresponding to each display feature in the first set of display features to obtain a first set of effective modalities; perform conflict modality fusion on the first set of effective modalities, and output a first effective modality combination.

[0009] Optionally, a trained modality correlation calculation model is embedded in the modality mapping engine; calculate multiple correlation index values corresponding to any display feature and multiple modality components according to the modality correlation calculation model; select effective modalities with a correlation index value greater than a preset correlation index value based on the multiple correlation index values, and so on, to obtain a set of effective modalities for each display feature, and output a first set of effective modalities.

[0010] Optionally, the VR system further includes a modality parser, and the modality parser is used to respectively receive a first set of sensing signals output by the first set of sensing devices; perform structured data parsing on the first set of sensing signals, and output a first set of effective modality data.

[0011] Optionally, collect a first set of interactive behavior feedback data corresponding to the first set of sensing devices when the user is in the virtual reality environment. Among them, the interactive behavior feedback data corresponding to each sensing device includes a modality response delay time, a number of operation failures, a modality usage frequency, and an action offset; input the first set of interactive behavior feedback data into a modality adjustment factor calculation model, and perform modality adjustment on the first effective modality combination according to the modality adjustment factor calculation model, and output a strengthened modality combination and a weakened modality combination; update and display the three-dimensional display content corresponding to the first set of effective modality data according to the strengthened modality combination and the weakened modality combination.

[0012] Optionally, initialize a modal adjustment factor calculation model according to a strengthening factor and a weakening factor, where the strengthening factor and the weakening factor are obtained through definition by a normalization activation function; calculate adjustment factors for the first effective modal combination respectively according to the modal adjustment factor calculation model, and output a first adjustment factor combination; output the factors greater than or equal to a preset threshold in the first adjustment factor combination as a strengthened modal combination; output the factors less than the preset threshold in the first adjustment factor combination as a weakened modal combination.

[0013] Optionally, when the VR system loads a second display theme, obtain a second effective modal combination and a second effective modal data set corresponding to the second display theme; load a corresponding second set of sensing devices according to the second effective modal combination, and the VR system switches from the first set of sensing devices to the second set of sensing devices.

[0014] In a second aspect, the present application further provides a thematized display system based on virtual reality for performing the thematized display method based on virtual reality described in the first aspect, where the thematized display system based on virtual reality includes: a theme recognition module for connecting to the VR system to load a theme configuration file and recognize a first display feature set of a first display theme in the theme configuration file; a modal requirement analysis module for connecting to a modal mapping engine and performing modal requirement analysis on the first display feature set according to the modal mapping engine to determine a first effective modal combination corresponding to the first display theme; an interaction event collection module for the VR system to control and load a corresponding first set of sensing devices from all sensing devices according to the first effective modal combination, and use the first set of sensing devices to collect interaction events in the virtual reality environment of the VR system, and output a first effective modal data set corresponding to the first effective modal combination; a display content generation module for generating three-dimensional display content corresponding to the first effective modal data set and displaying the three-dimensional display content in the virtual reality environment.

[0015] In a third aspect, the present application further provides an electronic device, including: at least one processor; a memory communicatively connected to the at least one processor; where the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the thematized display method based on virtual reality described in any one of the first aspects above.

[0016] One or more technical solutions provided in the present application have at least the following beneficial effects:

[0017] By connecting to the VR system to load the theme configuration file, identify the first set of display features of the first display theme in the theme configuration file; connect to the modality mapping engine, perform modality requirement analysis on the first set of display features according to the modality mapping engine, and determine the first valid modality combination corresponding to the first display theme; the VR system controls the loading of the corresponding first group of sensing devices from all sensing devices according to the first valid modality combination, uses the first group of sensing devices to collect interaction events in the virtual reality environment of the VR system, and outputs the first valid modality dataset corresponding to the first valid modality combination; generate the three-dimensional display content corresponding to the first valid modality dataset, and display the three-dimensional display content in the virtual reality environment. That is to say, by using virtual reality devices to collect the display theme information where the user is currently located, identifying the display theme according to multi-modal connection, extracting important display features associated with the theme, determining the interaction modality, loading and presenting immersive display content of the corresponding theme according to the interaction modality, and supporting real-time interaction feedback, the adaptability of theme display is improved, thereby enhancing the user experience and immersion.

[0018] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the following specifically gives the specific embodiments of the present application. It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only exemplary, and for those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0020] Figure 1 It is a flowchart of the method for theme-based display based on virtual reality of the present application.

[0021] Figure 2 It is a structural diagram of the theme-based display system based on virtual reality of the present application.

[0022] Figure 3 It is a structural diagram of an exemplary electronic device of the present application.

[0023] Description of the drawing reference numerals: The theme recognition module 11, the modal requirement analysis module 12, the interaction event collection module 13, the display content generation module 14, the bus 300, the receiver 301, the processor 302, the transmitter 303, the memory 304, the bus interface 305. Detailed implementation manners

[0024] By providing a themed display method, system, and electronic device based on virtual reality, the present application solves the technical problem in the prior art that due to the fixed virtual reality modal interaction method, it is difficult to adapt to the differentiated requirements of different display themes. By collecting the display theme information where the user is currently located through a virtual reality device, identifying the display theme according to multi-modal connection, extracting important display features associated with the theme, determining the interaction modality, loading and presenting the corresponding theme immersive display content according to the interaction modality, and supporting real-time interaction feedback, the adaptability of theme display is improved, thereby enhancing the user experience and immersion.

[0025] Next, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited by the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application. Additionally, it should be noted that for the sake of description, only the parts related to the present application are shown in the accompanying drawings rather than all of them.

[0026] Embodiment 1. Please refer to the attached Figure 1 , the present application provides a themed display method based on virtual reality. Among them, the themed display method based on virtual reality is executed by a themed display system based on virtual reality. The themed display method based on virtual reality specifically includes the following steps:

[0027] S100: Connect to the VR system to load the theme configuration file, and identify the first display feature set of the first display theme in the theme configuration file.

[0028] Further, S100 of the present application includes:

[0029] Identify the first display feature set of the first display theme in the theme configuration file. Among them, the fields of each theme configuration file include the theme number, theme type, and display content; read the fields of the theme configuration file, and parse the key display features of the first display theme based on NLP technology, and output them as the first display feature set. The key display features include physical entities, action elements, and interaction nodes.

[0030] Specifically, the VR system is connected, including the head-mounted display, interactive devices, content engine, theme configuration files, and interactive logic management platform. The theme configuration file is a structured configuration document that contains information about multiple VR display themes. Each theme includes its theme number, theme type, and display content. Each theme has its own unique display characteristics, such as visual style, sound effects, and interactive elements. For example, the theme profiles include Fingertip Sky (introducing the 20 common constellations, including an overview of the constellations, deep-space objects, personality traits, mythological stories, etc.), Weightlessness Diary (introducing the space environment, space communications, space life and space experiments, and displaying a virtual space modeled after the International Space Station), Space Climbing (introducing the outer space environment, spacewalks, and space maintenance-related content, creating a virtual external environment of a space station), Moon Racing (introducing impressions of the Moon Palace, lunar surface perception, lunar energy treasures, lunar journeys to Earth, and lunar base-related content, and building a realistic lunar surface scene), Brave Mars (truly restoring the process from launch from Earth to landing on Mars), Interstellar Travel (using Earth as a reference to understand the basic characteristics of the eight planets in the solar system), Deep Sea Dragon (experience all-round ocean exploration in a submersible cabin), and Three Lines of Defense (allowing the audience to experience the entire process of influenza from infection to recovery in a simulated interactive experience).

[0031] Among the multiple themes listed in the configuration file, the user's current display theme is collected and used as the first display theme. For example, the user selects the "Fingertip Sky" theme through the VR platform. The first display theme refers to the first display content that is automatically loaded or actively selected by the user when the user first enters the VR system. At the same time, the first display feature set corresponding to the first display theme is identified.

[0032] Each VR display theme is defined through a structured configuration file, which includes a theme number, theme type, and display content. The theme number is a unique identifier for quickly locating the theme content. The theme type, such as space exploration, marine science, or biological structure, facilitates categorization and management. Display content refers to the interactive content, objectives, and scenario requirements for the theme, described in text, annotations, and descriptive language. For example, the Fingertip Sky theme is Deep Space Series 01, and the theme type is MR mixed reality technology. The display content introduces the 20 common constellations (including the twelve constellations), including an overview of the constellations, deep sky objects, personality traits, and mythological stories. Using a hybrid virtual and real experience, the experience uses real data to recreate the full image of the corresponding celestial body. The user uses a controller to select a constellation and analyze it, experiencing the legend of space and exploring the mysteries of the starry sky. The Moon Rover theme is Deep Space Series 04, and the theme type is VR driving guidelines. The display content includes building a lunar rover, experiencing the lunar low-gravity driving experience, and simulating lunar surface data to gain a comprehensive understanding of lunar knowledge.

[0033] Extract field information from the corresponding theme configuration file: theme number, theme type, display content, and other textual description data. Use a pre-trained natural language processing engine to analyze the key display features of the first display theme, including physical entities (such as stars), action elements (such as climbing movements), and interactive nodes (such as virtual handrails and control panel buttons). Natural language processing (NLP) is a core technology for understanding and parsing natural language (such as text descriptions and voice input). It is used to automatically extract key information (feature elements) from the text fields in the configuration file, rather than relying on fixed templates or manual annotation.

[0034] By automatically parsing theme configuration files through NLP technology, users are provided with a rich and diverse virtual reality experience. Each theme has its own unique visual, auditory and interactive elements, which increases the attractiveness and fun of the display. At the same time, it can be easily applied to different display themes and has good scalability and flexibility.

[0035] S200: Connecting to a modality mapping engine, performing a modality requirement analysis on the first display feature set according to the modality mapping engine, and determining a first valid modality combination corresponding to a first display theme.

[0036] Furthermore, the present application S200 includes:

[0037] The modal mapping engine encapsulates multiple modal components, which are connected to multiple sensing devices, wherein the multiple modal components include at least a visual guidance modality, a force feedback modality, a motion capture modality, an audio modality, and a spatial navigation modality; the modal mapping engine maps and analyzes the required modalities corresponding to each display feature in the first display feature set to obtain a first valid modal set; and performs conflicting modal fusion on the first valid modal set to output a first valid modal combination.

[0038] Furthermore, the present application further comprises the following steps:

[0039] The modal mapping engine embeds a trained modal correlation calculation model; multiple correlation indicators corresponding to any display feature and multiple modal components are calculated according to the modal correlation calculation model; based on the multiple correlation indicators, a valid modality greater than a preset correlation indicator is selected, and so on, to obtain a valid modal set for each display feature and output a first valid modal set.

[0040] Specifically, the modal mapping engine is responsible for mapping various display features in the virtual reality display to corresponding perceptual modalities, such as vision, hearing, touch, motion, etc. Multiple available modal components are encapsulated within the modal mapping engine, including visual guidance modality, force feedback modality, motion capture modality, audio modality, and spatial navigation modality. Multiple modal components are connected to multiple perceptual devices. For example, the visual guidance modality component is connected to a head-mounted display, the force feedback modality component is connected to a force feedback glove, the motion capture modality is connected to an optical tracking device, and the audio modality is connected to a microphone array and spatial audio.

[0041] Based on each display feature in the first display feature set, the modal mapping engine performs mapping analysis to map it to the corresponding required modality, obtaining the first effective modality set. The modal mapping engine embeds a trained modal correlation calculation model, which calculates multiple correlation index values corresponding to multiple modal components according to any display feature in the display feature set. Through learning from historical interaction data, the modal correlation calculation model can output one or more numerical correlation index values, indicating the adaptability of a certain feature to a certain modality. The correlation index value usually ranges from 0 to 1, representing the matching strength between a display feature (such as grasping a rod) and a certain modal component (such as force feedback). The higher the index, the more suitable the modality is for supporting this feature.

[0042] When initializing the mapping engine, load the pre-trained model (which can be trained based on a deep neural network) into memory for real-time inference. For each feature description in the first display feature set, call the correlation model to sequentially calculate the matching scores between this feature and all modal components (visual guidance, force feedback, motion capture, audio, spatial navigation, etc.) in the engine; output a list of correlation index values corresponding to a set of features and modalities.

[0043] The preset correlation index is to pre-define a correlation threshold for screening sufficiently matching modal components. Only when the correlation index of a certain feature with a certain modality exceeds this threshold, this modality is considered an effective modality. Compare each set of correlation index values with the preset correlation index, and only retain the effective modalities with a correlation greater than or equal to the preset correlation index to form the effective modality set for each feature. Merge the effective modality sets corresponding to all display features to obtain the first effective modality set.

[0044] When multiple effective modalities are called in parallel, conflicts may occur due to resource contention or user perception interference. Therefore, conflict modality fusion is required to coordinate the modalities in order to generate a final interactive modality combination that not only meets the requirements of each feature but also avoids mutual interference. Determine which modalities have resource contention (such as the same visual rendering channel, the same tactile device channel) or perceptual logic conflicts (such as visual highlighting and spatial navigation simultaneously highlighting the same area, which will cause excessive visual burden). Mark all conflict pairs. For each pair of conflicting modalities, perform conflict modality fusion by means of priority assignment, time sequence staggering, modality substitution, strength adjustment, etc., and rearrange, delete, or adjust the parameters of all modality calls. Finally, output a conflict-free and structured first effective modality combination. For example, assume that the first effective modality set includes a visual guidance modality, a force feedback modality, and a motion capture modality. If there is a conflict between the visual guidance modality and the motion capture modality, choose to reduce the intensity of the visual guidance modality to reduce the competition for the user's attention, thereby obtaining the best effective modality combination.

[0045] Among them, priority assignment refers to setting the order of precedence for conflicting modalities according to the importance of the display features or user preferences; time sequence staggering refers to staggering the call times of conflicting modalities. For example, first execute visual guidance and then execute spatial navigation; modality substitution refers to replacing with a backup modality with similar functions when a certain modality device or channel is incompatible; strength adjustment refers to weakening the output of the modality with lower priority within the same time window (such as reducing the visual highlighting intensity or audio volume).

[0046] Exemplarily, assume that the display theme is space climbing. Calculate the multiple correlation degree indicators corresponding to any display feature and multiple modality components as follows: the correlation degree indicator between the climbing action and the visual guidance modality is 0.88, the correlation degree indicator between the climbing action and the force feedback modality is 0.92, the correlation degree indicator between the climbing action and the motion capture modality is 0.95, the correlation degree indicator between the climbing action and the spatial navigation modality is 0.5, and the correlation degree indicator between the climbing action and the audio modality is 0.56. The preset correlation degree threshold is 0.8. Then, the visual guidance modality, the force feedback modality, and the motion capture modality are selected as the effective modalities for this feature.

[0047] By means of the modality correlation degree calculation model, selecting appropriate effective modalities according to the correlation degree between the display features and the modality components helps to improve the user's immersion and interactive experience in the virtual reality environment, making the display more vivid and realistic.

[0048] S300: The VR system controls the loading of the corresponding first perception device group from all perception devices according to the first effective modality combination, uses the first perception device group to collect interaction events in the virtual reality environment of the VR system, and outputs the first effective modality data set corresponding to the first effective modality combination.

[0049] Furthermore, step S300 of this application includes:

[0050] The VR system further includes a modality parser, which is configured to receive the first set of perception signals output by the first group of perception devices respectively; perform structured data parsing on the first set of perception signals, and output a first effective modality dataset.

[0051] Specifically, the VR system loads the corresponding first group of perception devices from all perception devices according to the first effective modality combination, such as visual modality (display screen, eye tracker), auditory modality (headphones, microphone), tactile modality (vibrating gloves, force feedback device), action modality (body sensor camera, motion capture device). For example, if the first effective modality combination includes visual guidance modality, force feedback modality and motion capture modality, then the first group of perception devices may include a display screen, an eye tracker, vibrating gloves and a motion capture device. Interaction event collection is performed in the virtual reality environment of the VR system through the first group of perception devices, that is, when the VR system runs in the environment, various operations and physiological reactions of the user are recorded by the perception devices, such as eye fixation points, voice commands, gesture actions, body postures, etc. For example, when the user interacts with a virtual character in the virtual reality environment, the eye tracker will record the user's eye movement, the vibrating gloves will provide tactile feedback, and the motion capture device will capture the user's actions.

[0052] The VR system includes a modality parser, which is responsible for receiving the original signals from each perception device (such as image stream, audio stream, tactile sensing value, action coordinate data), and uniformly parsing these heterogeneous inputs with different protocols into a structured data format, that is, the first effective modality dataset. Specifically, the stream data collected by each device is aligned in time sequence according to the time stamp, and structured data parsing is performed relying on their respective parsing algorithms. The visual data outputs the fixation hot zone through the fixation trajectory analysis module; the audio stream is transcribed into a text command through the speech recognition module; the tactile sensing value is converted into an event trigger mark; the action coordinate data is mapped into a standard action event through the coordinate conversion algorithm. The multi-modal signals are encapsulated into the first effective modality dataset, which contains the processed and organized interaction data corresponding to the first effective modality combination, including structured data such as fixation point coordinates, hand joint angles and contact forces, and is used to generate three-dimensional display content.

[0053] By loading and utilizing the first group of perception devices, the VR system can perform interaction event collection in the virtual reality environment and generate the first effective modality dataset, which helps to improve the immersion and interaction experience of the user in the virtual reality environment, makes the display more vivid and real, provides accurate visual, tactile and action feedback, thereby enhancing the realism and interactivity of the user experience.

[0054] S400: Generate three-dimensional display content corresponding to the first effective modal dataset and display the three-dimensional display content in the virtual reality environment.

[0055] Further, S400 of the present application includes:

[0056] Collect a first set of interactive behavior feedback data corresponding to the first perception device group of the user in the virtual reality environment. Among them, the interactive behavior feedback data corresponding to each perception device includes modal response delay time, number of operation failures, modal usage frequency, and action offset; input the first set of interactive behavior feedback data into a modal adjustment factor calculation model, and perform modal adjustment on the first effective modal combination according to the modal adjustment factor calculation model, and output a strengthened modal combination and a weakened modal combination; update and display the three-dimensional display content corresponding to the first effective modal dataset according to the strengthened modal combination and the weakened modal combination.

[0057] Further, the present application further includes the following steps:

[0058] Initialize a modal adjustment factor calculation model according to a strengthening factor and a weakening factor, where the strengthening factor and the weakening factor are obtained by defining through a normalization activation function; perform adjustment factor calculation on the first effective modal combination according to the modal adjustment factor calculation model, and output a first set of adjustment factors; output the factors in the first set of adjustment factors that are greater than or equal to a preset threshold as a strengthened modal combination; output the factors in the first set of adjustment factors that are less than the preset threshold as a weakened modal combination.

[0059] Specifically, obtain a first set of interactive behavior feedback data generated when the user interacts through the first perception device group in the virtual reality environment, including the interactive behavior feedback data corresponding to each perception device, such as modal response delay time, number of operation failures, modal usage frequency, and action offset. The modal response delay time is the time difference between when the user performs an interactive operation and when the perception device gives a response; the number of operation failures is the number of times the user fails in the interaction process; the modal usage frequency is the frequency of the user using a certain modal in the interaction process; the action offset is the deviation between the actual action and the expected action of the user in the interaction process.

[0060] Input the interaction behavior feedback data (such as modal response delay time, number of operation failures, modal usage frequency, and action offset) into the normalization activation function module. Apply the Sigmoid mapping to the original evaluation value of each modality, limit the value within the range of 0 to 1, and use it as the initial reinforcement / weakening factor. Specifically, calculate the ratios of the modal response delay time, number of operation failures, modal usage frequency, and action offset to their respective maximum limit values, and then adjust these values through an adjustment coefficient. Input these values into the normalization activation function (such as the sigmoid function) to obtain the reinforcement factor and the weakening factor. The reinforcement factor and the weakening factor respectively represent the numerical weights indicating the enhancement or weakening trend of a certain interaction modality output in the modal adjustment factor calculation model.

[0061] Initialize the modal adjustment factor calculation model with the reinforcement factor and the weakening factor. Calculate the adjustment factors for the first valid modal combination respectively to obtain the first adjustment factor combination, which contains the adjustment factors of each modality and reflects the degree to which they should be enhanced or weakened. When the task is not completed, the feature priority is high, and the modal feedback is weak, the modal change is strengthened; when the task is approaching completion or there is a lot of modal feedback and the response is slow, the modal change is weakened.

[0062] Compare each adjustment factor with a preset threshold. This threshold range is used to classify the adjustment factors into three categories: need to be strengthened, maintained, or weakened. When the first adjustment factor combination is greater than or equal to the maximum value of the preset threshold, increase the manifestation frequency / brightness / volume / sensitivity, etc. of this modality, and the output is the strengthened modal combination; when the first adjustment factor combination is less than the minimum value of the preset threshold, reduce the manifestation intensity of this modality or hide some content, and the output is the weakened modal combination; when the first adjustment factor combination is within the threshold range, maintain the current state and wait for further interaction confirmation. The strengthened modal combination is the modality with high factors and good performance, and the manifestation intensity needs to be increased in rendering or interaction (such as brighter highlights, stronger vibrations, louder reminder sounds); the weakened modal combination is the modality with low factors, poor performance, or causing interference, and the manifestation priority needs to be reduced or the reminder frequency needs to be decreased.

[0063] Update and display the three-dimensional display content corresponding to the first valid modal dataset according to the strengthened modal combination and the weakened modal combination, which helps to improve the immersion and interaction experience of the user in the virtual reality environment and makes the display more vivid and realistic. For the strengthened modality, increase the brightness, frequency, or intensity of the corresponding reminder in the three-dimensional scene; for the weakened modality, reduce the reminder intensity or temporarily hide the relevant interaction elements during non-critical periods. For example, if the strengthened modal combination includes the force feedback modality, then the VR system increases the intensity of the force feedback to make the interaction of the user in the virtual reality environment more realistic. If the weakened modal combination includes the visual guidance modality, then the VR system reduces the intensity of the visual guidance to reduce the competition for the user's attention.

[0064] Based on the first effective modal dataset, corresponding three-dimensional display content is generated, including the three-dimensional model of the scene, textures, animations, sounds, force feedback, etc. Using a virtual reality rendering engine, the generated three-dimensional display content is rendered into a virtual reality environment, including real-time rendering of the three-dimensional model, mapping of textures and materials, playing of animations, processing of sounds, and simulation of force feedback, etc. Through interactive devices such as VR headsets, headphones, and controllers, the rendered three-dimensional display content is presented to the user. The user can freely move, observe, and interact in the virtual reality environment to obtain an immersive experience. According to the user's interaction behaviors and feedback, the display intensity or frequency of the three-dimensional display content is dynamically adjusted to optimize the user experience. Finally, the rendered three-dimensional image is presented through a head-mounted display, the sound is played through headphones, and tactile and motion feedback are synchronized to create a complete multi-modal immersive scene.

[0065] By collecting the interaction behavior feedback data of the user in the virtual reality environment and using the modal adjustment factor calculation model to adjust the effective modal combination, the VR system can optimize the three-dimensional display content according to the actual feedback of the user, which helps to improve the immersion and interaction experience of the user in the virtual reality environment and makes the display more vivid and real.

[0066] Furthermore, the present application further includes the following steps:

[0067] When the VR system loads the second display theme, obtain the second effective modal combination and the second effective modal dataset corresponding to the second display theme; according to the second effective modal combination, load the corresponding second group of perception devices, and the VR system switches from the first group of perception devices to the second group of perception devices.

[0068] Specifically, when the VR system loads the second display theme, obtain the second display feature set corresponding to the second display theme, and determine the second effective modal combination corresponding to the second display theme according to the modal mapping engine. In the second display theme scene, the modal parser collects the perception signals output by the second group of perception devices and performs structured data parsing to obtain the second effective modal dataset.

[0069] Stop and disconnect the sensing devices used for the first theme (such as deactivating gesture capture, haptic gloves, or eye trackers), and release the hardware resources; Dynamically start and connect new sensing devices according to the second modality combination (such as enabling an infrared camera if the new theme requires infrared tracking, or starting surround sound headphones if ambient sound is needed); Ensure that the new devices complete calibration and initialization in the background to ensure seamless connection. After the switch is completed, immediately display the basic 3D environment in the new theme scene and trigger an initial interaction prompt to confirm that the second sensing device group is working properly. The second sensing device group is a set of hardware devices dynamically loaded according to the second effective modality combination, which may partially overlap with the first sensing device group or may be completely different to optimally support the interaction requirements of the second theme.

[0070] Dynamically switch the sensing device group according to different display themes, providing the corresponding immersion and interaction experience. The sensing device group can be quickly switched and reconfigured between different themes, increasing the flexibility and adaptability of the VR system and meeting the requirements of different display themes.

[0071] In summary, the themed display method based on virtual reality provided by this application has the following beneficial effects:

[0072] By connecting the VR system to load the theme configuration file, identify the first display feature set of the first display theme in the theme configuration file; Connect the modality mapping engine, and perform modality requirement analysis on the first display feature set according to the modality mapping engine to determine the first effective modality combination corresponding to the first display theme; The VR system controls the loading of the corresponding first sensing device group from all sensing devices according to the first effective modality combination, and uses the first sensing device group to collect interaction events in the virtual reality environment of the VR system, and outputs the first effective modality data set corresponding to the first effective modality combination; Generate the three-dimensional display content corresponding to the first effective modality data set, and display the three-dimensional display content in the virtual reality environment. That is to say, by using virtual reality devices to collect the display theme information where the user is currently located, identify the display theme according to multimodal connection, extract the important display features associated with the theme, determine the interaction modality, load and present the immersive display content of the corresponding theme according to the interaction modality, and support real-time interaction feedback, improving the adaptability of theme display, thereby enhancing the user experience and immersion.

[0073] Embodiment 2, based on the same inventive concept as the themed display method based on virtual reality in the first embodiment above, this application also provides a themed display system based on virtual reality. Please refer to the appendix Figure 2 The themed display system based on virtual reality includes:

[0074] A theme recognition module 11, which is used to connect to the VR system to load a theme configuration file and recognize a first display feature set of a first display theme in the theme configuration file; a modality requirement analysis module 12, which is used to connect to a modality mapping engine and perform modality requirement analysis on the first display feature set according to the modality mapping engine to determine a first effective modality combination corresponding to the first display theme; an interaction event collection module 13, which is used for the VR system to control and load a corresponding first perception device group from all perception devices according to the first effective modality combination, collect interaction events in the virtual reality environment of the VR system by using the first perception device group, and output a first effective modality data set corresponding to the first effective modality combination; a display content generation module 14, which is used to generate three-dimensional display content corresponding to the first effective modality data set and display the three-dimensional display content in the virtual reality environment.

[0075] Furthermore, the theme recognition module 11 in the theme-based display system based on virtual reality is further used for:

[0076] Recognize a first display feature set of a first display theme in the theme configuration file, where each field of the theme configuration file includes a theme number, a theme type, and display content; read the fields of the theme configuration file, and parse the key display features of the first display theme based on NLP technology as the first display feature set for output, and the key display features include physical entities, action elements, and interaction nodes.

[0077] Furthermore, the modality requirement analysis module 12 in the theme-based display system based on virtual reality is further used for:

[0078] A plurality of modality components are encapsulated in the modality mapping engine, and the plurality of modality components are connected to a plurality of perception devices, where the plurality of modality components at least include a visual guidance modality, a force feedback modality, an action capture modality, an audio modality, and a spatial navigation modality; the modality mapping engine maps and analyzes the required modalities corresponding to each display feature in the first display feature set to obtain a first effective modality set; perform conflict modality fusion on the first effective modality set, and output a first effective modality combination.

[0079] Furthermore, the modality requirement analysis module 12 in the theme-based display system based on virtual reality is further used for:

[0080] The modality mapping engine embeds a trained modality correlation calculation model; calculates a plurality of correlation index values corresponding to any display feature and a plurality of modality components according to the modality correlation calculation model; selects effective modalities greater than a preset correlation index value according to the plurality of correlation index values, and so on, to obtain an effective modality set for each display feature, and outputs a first effective modality set.

[0081] Furthermore, the interaction event collection module 13 in the virtual reality-based themed display system is further configured to:

[0082] The VR system further includes a modality parser, which is configured to respectively receive a first set of perception signals output by the first set of perception devices; perform structured data parsing on the first set of perception signals, and output a first effective modality data set.

[0083] Furthermore, the display content generation module 14 in the virtual reality-based themed display system is further configured to:

[0084] Collect a first set of interactive behavior feedback data corresponding to the first set of perception devices of the user in the virtual reality environment, where the interactive behavior feedback data corresponding to each perception device includes a modality response delay time, a number of operation failures, a modality usage frequency, and an action offset; input the first set of interactive behavior feedback data into a modality adjustment factor calculation model, perform modality adjustment on the first effective modality combination according to the modality adjustment factor calculation model, and output a strengthened modality combination and a weakened modality combination; update and display the three-dimensional display content corresponding to the first effective modality data set according to the strengthened modality combination and the weakened modality combination.

[0085] Furthermore, the display content generation module 14 in the virtual reality-based themed display system is further configured to:

[0086] Initialize a modality adjustment factor calculation model according to a strengthening factor and a weakening factor, where the strengthening factor and the weakening factor are obtained by defining a normalization activation function; perform adjustment factor calculation on the first effective modality combination according to the modality adjustment factor calculation model, and output a first set of adjustment factors; output the factors greater than or equal to a preset threshold in the first set of adjustment factors as a strengthened modality combination; output the factors less than the preset threshold in the first set of adjustment factors as a weakened modality combination.

[0087] Furthermore, the virtual reality-based themed display system further includes a second theme module, and the second theme module is further configured to:

[0088] When the VR system loads a second display theme, obtain a second effective modality combination and a second effective modality data set corresponding to the second display theme; load a corresponding second set of perception devices according to the second effective modality combination, and the VR system switches from the first set of perception devices to the second set of perception devices. Each embodiment in this specification is described in a progressive manner, and the key point of each embodiment is the difference from other embodiments. The foregoing Figure 1The themed display method and specific examples in the first embodiment based on virtual reality are equally applicable to the themed display system based on virtual reality in this embodiment. Through the detailed description of the themed display method based on virtual reality above, those skilled in the art can clearly know the themed display system based on virtual reality in this embodiment. Therefore, for the sake of simplicity of the specification, it will not be elaborated here.

[0089] Embodiment 3, based on the same inventive concept as the themed display method based on virtual reality in the first embodiment above, the present application also provides an electronic device, including: at least one processor; a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps of the themed display method based on virtual reality according to any one of the above-mentioned first embodiments.

[0090] Appendix Figure 3 is a schematic structural diagram of an exemplary electronic device of the present application. In Figure 3 , the bus architecture is represented by bus 300. Bus 300 may include any number of interconnected buses and bridges. Bus 300 connects various circuits including one or more processors represented by processor 302 and a memory represented by memory 304 together. Bus 300 may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art. Therefore, they will not be further described herein. Bus interface 305 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 302 is responsible for managing bus 300 and general processing, while memory 304 may be used to store data used by processor 302 when performing operations.

[0091] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0092] Obviously, for those skilled in the art, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A themed display method based on virtual reality, characterized in that Including: Connect to the VR system to load the theme configuration file, and identify the first display feature set of the first display theme in the theme configuration file; Connect to the modality mapping engine, perform modality requirement analysis on the first display feature set according to the modality mapping engine, and determine the first effective modality combination corresponding to the first display theme; The VR system controls the loading of the corresponding first perception device group from all perception devices according to the first effective modality combination, uses the first perception device group to collect interaction events in the virtual reality environment of the VR system, and outputs the first effective modality data set corresponding to the first effective modality combination; Generate the three-dimensional display content corresponding to the first effective modality data set, and display the three-dimensional display content in the virtual reality environment; Identify the first display feature set of the first display theme in the theme configuration file, where the fields of each theme configuration file include theme number, theme type, and display content; Read the fields of the theme configuration file, parse the key display features of the first display theme based on NLP technology, and output them as the first display feature set. The key display features include physical entities, action elements, and interaction nodes; Perform modality requirement analysis on the first display feature set according to the modality mapping engine, including: Multiple modality components are encapsulated in the modality mapping engine, and the multiple modality components are connected to multiple perception devices. Among them, the multiple modality components at least include visual guidance modality, force feedback modality, motion capture modality, audio modality, and spatial navigation modality; The modality mapping engine maps and analyzes the required modalities corresponding to each display feature in the first display feature set to obtain the first effective modality set; Perform conflict modality fusion on the first effective modality set, and output the first effective modality combination.

2. The themed display method based on virtual reality according to claim 1, wherein The method for obtaining the first effective modality set includes: The modality mapping engine embeds a trained modality correlation calculation model; Calculate multiple correlation index values corresponding to any display feature and multiple modality components according to the modality correlation calculation model; Select the effective modalities with a correlation index value greater than the preset correlation index value according to the multiple correlation index values, and so on, to obtain the effective modality sets of each display feature, and output the first effective modality set.

3. The themed display method based on virtual reality according to claim 1, wherein The VR system further includes a modality parser, and the modality parser is used to respectively receive the first perception signal group output by the first perception device group; Perform structured data parsing on the first perception signal group, and output the first effective modality data set.

4. The themed display method based on virtual reality according to claim 1, wherein, After displaying the three-dimensional display content in the virtual reality environment, it further includes: Collect the first interaction behavior feedback data group of the user in the virtual reality environment based on the first perception device group. Among them, the interaction behavior feedback data corresponding to each perception device includes modality response delay time, number of operation failures, modality usage frequency, and action offset; Input the first interaction behavior feedback data group into the modality adjustment factor calculation model, perform modality adjustment on the first effective modality combination according to the modality adjustment factor calculation model, and output the enhanced modality combination and the weakened modality combination; Update and display the three-dimensional display content corresponding to the first effective modality dataset according to the described enhanced modality combination and weakened modality combination.

5. The themed display method based on virtual reality according to claim 4, characterized in that Perform modality adjustment on the first effective modality combination according to the modality adjustment factor calculation model, including: Initialize the modality adjustment factor calculation model according to the enhancement factor and the weakening factor, where the enhancement factor and the weakening factor are obtained through definition by a normalization activation function; Perform adjustment factor calculation on the first effective modality combination respectively according to the modality adjustment factor calculation model, and output a first adjustment factor combination; Output the factors greater than or equal to the preset threshold in the first adjustment factor combination as the enhanced modality combination; Output the factors less than the preset threshold in the first adjustment factor combination as the weakened modality combination.

6. The themed display method based on virtual reality according to claim 1, wherein Connect the VR system to load the theme configuration file, further including: When the VR system loads the second display theme, obtain the second effective modality combination and the second effective modality dataset corresponding to the second display theme; Load the corresponding second perception device group according to the second effective modality combination, and the VR system switches from the first perception device group to the second perception device group.

7. A themed display system based on virtual reality, characterized in that, For implementing the steps of the virtual reality-based themed display method according to any one of claims 1 to 6, the virtual reality-based themed display system includes: A theme recognition module, configured to connect the VR system to load the theme configuration file and recognize the first display feature set of the first display theme in the theme configuration file; A modality requirement analysis module, configured to connect the modality mapping engine and perform modality requirement analysis on the first display feature set according to the modality mapping engine to determine the first effective modality combination corresponding to the first display theme; An interaction event acquisition module, configured to control the VR system to load the corresponding first perception device group from all perception devices according to the first effective modality combination, and use the first perception device group to perform interaction event acquisition in the virtual reality environment of the VR system, and output the first effective modality dataset corresponding to the first effective modality combination; A display content generation module, configured to generate the three-dimensional display content corresponding to the first effective modality dataset and display the three-dimensional display content in the virtual reality environment.

8. An electronic device, characterized in that, Including: At least one processor; A memory communicatively connected to the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the virtual reality-based themed display method according to any one of claims 1 to 6.

Images