Cloud VR immersive experience system

Abstract

The invention discloses a cloud VR immersive experience system, and relates to the technical field of virtual reality, the system comprises the following components: a cloud server module, a user terminal module, a tactile feedback module and an interaction module; the cloud server module comprises a hardware structure and a software structure, the hardware structure comprises a high-performance server cluster, and the touch feedback module is integrated, so that the real touch of a user in VR experience is remarkably enhanced; wherein the tactile feedback script analysis unit can flexibly analyze and apply a complex tactile feedback script written by a developer, and the tactile effect generation unit can accurately control advanced tactile feedback equipment to simulate various real tactile senses including hardness, roughness and temperature, so that the immersive experience of a user is greatly enriched, and the user experience is improved. And the user can feel more real and exquisite physical interaction in the virtual world, so that the overall VR experience quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a cloud-based VR immersive experience system. Background Art

[0002] With the rapid development of science and technology, virtual reality (VR) technology, as a new generation of human-computer interaction technology, is gradually penetrating into multiple fields such as entertainment, education, medical care, industrial design, etc., bringing users an unprecedented immersive experience. VR technology simulates the environment, perception, natural skills and sensing equipment, allowing users to interact with the virtual world as if they were in a three-dimensional environment generated by a computer. In recent years, with the rise of technologies such as WebVR and WebXR, VR experience has been more conveniently integrated into users' daily lives and is no longer limited to specific hardware platforms or devices.

[0003] Existing VR systems are often limited to specific hardware platforms or devices. Users need to purchase specific VR headsets, controllers and other equipment to experience VR content, which not only increases the financial burden on users, but also limits the popularization of VR technology. Secondly, most mobile VR applications require users to download and install corresponding software or games, which not only takes up the storage space of the user's device, but may also cause compatibility issues between different platforms.

[0004] In terms of immersive experience, although VR technology can already provide relatively realistic visual and auditory effects, there is still much room for improvement in tactile feedback. Traditional mobile VR systems often rely on the built-in vibrator of the mobile phone to provide simple tactile feedback. This feedback method is single and lacks realism, and it is difficult to simulate complex touch, such as hardness, roughness, temperature, etc. In addition, due to the limitations of mobile phone hardware resources, traditional mobile VR systems find it difficult to achieve high-quality real-time rendering and interaction, resulting in a user experience that is not smooth and realistic enough. At the same time, the creation of VR content for mobile platforms is also relatively complex, requiring developers to have a high technical threshold, which further limits the popularization and development of mobile VR technology. Summary of the invention

[0005] The purpose of the present invention is to make up for the shortcomings of the existing technology and provide a cloud-based VR immersive experience system, which can achieve seamless access and display of VR content on multiple devices by utilizing WebVR and WebXR technologies. The cloud server optimizes the VR content to ensure the adaptability and fluency of the content on devices with different performances. At the same time, the introduction of a tactile feedback module, combined with a customized tactile feedback script, provides users with a richer and more realistic tactile experience. In addition, the present invention also simplifies the creation process of VR content, lowers the technical threshold, and helps promote the popularization and application of VR technology on mobile phones.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a cloud-based VR immersive experience system, the system comprising the following components: a cloud-based server module, a user terminal module, a tactile feedback module and an interaction module;

[0007] The cloud server module includes a hardware structure and a software structure, wherein the hardware structure includes a high-performance server cluster, wherein the server cluster includes multiple computing nodes and storage nodes, which are interconnected through a high-speed network, and the software structure includes a data storage layer, a content processing layer, and a network communication layer, wherein the data storage layer uses a distributed file system to store VR content data, and establishes a database system to store VR content metadata, the content processing layer has a real-time rendering engine, and can render VR content in real time according to user interaction operations, and the network communication layer uses HTTP / 2 and higher versions of the protocol to communicate with user terminals, and establish an efficient network connection management mechanism;

[0008] The user terminal module can receive and parse VR content from the cloud server, and render and display it locally. The module also includes an interface connected to a tactile feedback device for receiving and processing tactile feedback information.

[0009] The tactile feedback module is connected to the user terminal module and is used to interact with the advanced tactile feedback device. The module includes a tactile feedback script parsing unit and a tactile effect generating unit. The tactile feedback script parsing unit receives the current VR application scene information and the pre-written tactile feedback script. The parsing engine parses the script according to the grammatical structure of the script, extracts the scene object information, the touch action description and the corresponding tactile effect parameters, including hardness value, roughness value and temperature value, and determines the specific parameter value of the tactile effect to be generated according to the extracted information and the current application scene. The tactile effect generating unit generates the tactile effect parameters for the mobile phone tactile feedback device according to the tactile effect parameters determined by the tactile feedback script parsing unit by obtaining the vibration coefficient. Combining formulas Calculate the vibration intensity, where V phome Indicates the vibration intensity of the phone's tactile feedback device. is the vibration coefficient of the mobile phone, H is the hardness value obtained by analyzing the tactile feedback script, H0 is the reference hardness value, It is the touch type adjustment factor in the mobile phone application scenario. is the correlation coefficient of the current state of the mobile phone, C phone Indicates the current available power or computing resource ratio of the mobile phone, C max Indicates the maximum value of the mobile phone power or computing resources, and according to the formula Calculate the vibration frequency, where f phome Indicates the vibration frequency of the mobile phone's tactile feedback device. is the frequency coefficient specific to the mobile phone, R is the roughness value obtained by parsing the tactile feedback script, and R0 is the base roughness value. is the roughness type adjustment factor, is the coefficient related to the mobile phone screen interaction, S interaction Indicates the proportion of the touch-related interactive area on the current screen, S total Indicates the total area of ​​the screen, and is used to dynamically adjust the vibration frequency according to the user's interactive position and area on the screen, to provide tactile feedback that is more in line with the user's operation. According to the calculated vibration intensity and vibration frequency, a control instruction is sent to the vibration motor tactile feedback device through the Bluetooth interface to control the device to generate a corresponding tactile sensation. During the tactile feedback process, the user's feedback information is continuously monitored. If the user feedback is that the tactile sensation is too strong or too weak, the tactile effect parameters are adjusted according to the user feedback. For the vibration motor tactile feedback device, the vibration intensity and vibration frequency are recalculated to optimize the tactile effect;

[0010] The interaction module is used to realize real-time interaction between the user and the VR virtual environment, and also includes a local interaction processing unit on the user terminal device for processing the interaction logic.

[0011] Furthermore, the cloud server module includes a real-time rendering engine, which can render VR content in real time according to the user's interactive operation, and determine the rendering order of objects by comprehensively considering the distance from the object to the observation point and the importance weight of the object. Specifically, for an object in a three-dimensional space, its position coordinate is (x i ,y i ,z i ), the observation point coordinates are (x0, y0, z0), then the distance Determine the importance weight w of the object based on scene design and interaction requirements i , calculate the comprehensive ranking value Among them, α is the weight coefficient. By adjusting the value of α, the influence of distance and importance weight on the ranking can be balanced.

[0012] Furthermore, the cloud server module includes a real-time rendering engine, which can render VR content in real time according to the user's interactive operation. Specifically, for each object to be rendered, the reflection coefficient k in its material information is obtained according to the order determined by the rendering sorting algorithm. r and the transmission coefficient k t parameters, and determine the incident light intensity L i , calculate the incident light intensity L according to the intensity of the light source in the illumination information table and the relative position and direction relationship between the object and the light source i , let the light source intensity be I0, and the distance from the light source to the object be d0, then the incident light intensity In addition, the reflected light intensity L r The calculation formula is: L r =k r ×L i ×F(θ i ,θ r ), where θ i is the angle of incidence, θ r is the reflection angle, F(θ i ,θ r ) is the Fresnel reflection function, and the incident angle and reflection angle are determined by the angle between the light and the normal line of the object surface, and the transmitted light intensity L is calculated t =k t ×L i ×T9θ i ,θ t ), where θ i is the angle of incidence, θ t is the transmission angle, T9θ i ,θ t ) is the transmission function. Similarly, the incident angle and transmission angle are determined by the relationship between the light and the surface normal of the object, and the final light intensity L = L is calculated. e +L r +L t , where L e is the self-luminous intensity of the object. For objects without self-luminous properties, L e =0.

[0013] Furthermore, the distributed file system of the data storage layer of the cloud server module adopts Ceph, the database system adopts MySQL and PostgreSQL, and the WebVR and WebXR technology related processing library of the content processing layer is used to convert the format and optimize the selected VR content to make it suitable for user terminal devices.

[0014] Furthermore, the user terminal module includes hardware support and software architecture, and its hardware support includes that the P device requires a memory of more than 16GB, the network interface is a Gigabit Ethernet interface or a high-speed Wi-Fi module, and the Android device and tablet require the processor to be an ARM Cortex-A series high-end processor, with a memory of more than 8GB and a screen resolution of 2K or higher.

[0015] Furthermore, the user terminal module includes hardware support and software architecture, and its software architecture includes a browser component and a device driver component. The browser component adopts a browser kernel that supports WebVR and WebXR technologies, and the kernel integrates an interface module for communicating with the tactile feedback device and establishes a local caching mechanism. The device driver component develops corresponding drivers for different input devices and tactile feedback devices.

[0016] Furthermore, the interaction module includes a hardware interface and a software architecture, wherein the hardware interface provides a variety of interfaces to connect input devices, including a USB interface, a Bluetooth interface and a sensor interface.

[0017] Furthermore, the interaction module software architecture includes an input device interface component and a local interaction processing unit. The input device interface component develops corresponding interface drivers for different input devices. The local interaction processing unit uses a rule engine to process simple interaction operations and maintains communication with the cloud server module. When encountering complex interaction operations, it sends instructions to the cloud server module.

[0018] Compared with the existing technology, this cloud-based VR immersive experience system has the following beneficial effects:

[0019] 1. The present invention significantly enhances the user's realistic sense of touch in VR experience by integrating a tactile feedback module. The tactile feedback script parsing unit can flexibly parse and apply complex tactile feedback scripts written by developers, while the tactile effect generation unit can accurately control advanced tactile feedback devices to simulate a variety of real touch sensations including hardness, roughness and temperature, greatly enriching the user's immersive experience and enabling the user to feel more realistic and delicate physical interactions in the virtual world, thereby improving the overall VR experience quality.

[0020] 2. The present invention uses WebVR and WebXR technologies to achieve seamless compatibility of VR experience on multiple user terminal devices, including PCs, Android devices and tablets. Users do not need to download and install additional applications or games. They can directly access VR content on the cloud server through the browser on the device, which greatly simplifies the usage process and lowers the user threshold. This cross-platform compatibility and convenience not only broadens the market coverage of VR experience, but also promotes the popularization and promotion of VR technology.

[0021] Other advantages, objectives and features of the present invention will be set forth in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1A schematic diagram of a cloud-based VR immersive experience system;

[0024] Figure 2 The figure is a flow chart of a cloud-based VR immersive experience system. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Embodiment 1

[0027] This embodiment describes in detail a specific application of a cloud-based VR immersive experience system in the field of mobile games.

[0028] First, the cloud server module includes hardware architecture and software architecture. For the hardware architecture, the cloud server adopts a high-performance server cluster, the computing nodes are equipped with multi-core processors and large-capacity memory, and the storage nodes use large-capacity disk arrays, which provide powerful computing and storage capabilities for processing complex VR game scene data. For the software architecture, it includes the data storage layer, content processing layer and network communication layer. The data storage layer uses the Ceph distributed file system to store the game's VR scene data (such as jungle terrain, tree models, animal models, etc.), object data (such as props, weapons, etc.) and interactive scripts (such as task scripts, character behavior scripts, etc.). At the same time, MySQL or PostgreSQL database systems are used to store the metadata of these data, including scene name, applicable device type, creation time and other information. The distributed file system and database system are associated through the index mechanism to quickly locate and obtain the required data. The content processing layer integrates WebV R and WebXR technology related processing libraries, such as the relevant parts of the Three.js library, etc., after receiving the performance parameters and game requests sent by the mobile phone, the dense jungle scene data suitable for the performance of the mobile phone is selected from the storage, and then the scene data is converted into a format suitable for mobile browser rendering using the processing library, such as converting the 3D model into the WebGL format, and optimizing the texture and model to improve rendering efficiency and quality. The real-time rendering engine uses ray tracing related algorithms to simulate the scene lighting effects. For the light source of sunlight in the jungle, its self-luminous intensity is set to a suitable value (representing the intensity of sunlight). Objects such as leaves and trunks have different reflection coefficients and transmission coefficients according to their own material characteristics. For example, leaves may have a higher reflection coefficient to simulate the reflection effect of their smooth surface, and trunks may have a lower reflection coefficient and a certain transmission coefficient to simulate the optical properties of their wooden structure. Through the lighting calculation formula L=Le +L r +L t Calculate the light intensity on the surface of each object, and the reflected light intensity L r , through the formula L r =k r ×L i ×F(θ i ,θ r ) calculation, where L i is the incident light intensity (calculated based on the distance from the sunlight to the object and the intensity attenuation law), F(θ i ,θ r ) is the Fresnel reflection function, and the incident angle θ is determined by the angle between the light and the normal line of the object surface. i and the reflection angle θ r , for the transmitted light intensity L t , through the formula L t =k t ×L i ×T(θ i ,θ t ) calculation, where T(θ i ,θ t ) is the transmission function, and the incident angle θ is also determined by the angle between the light and the normal line of the object surface. i and the transmission angle θ t At the same time, according to the scene object rendering sorting algorithm Sort the objects in the scene. In the jungle scene, α is set to 0.5 (balancing distance and importance weight). For objects that are closer to the player, such as trees and stones around the player, their distance to the observation point (player's perspective) is d. i Small, importance weight w i It may be higher (because the player pays more attention to the surrounding environment), so the comprehensive ranking value S i Smaller ones are rendered first to improve rendering efficiency. The network communication layer uses HTTP / 2 or higher versions of the protocol to communicate with mobile user terminals, and uses the protocol's multiplexing and header compression features to improve data transmission efficiency. At the same time, the transmitted data is compressed with GZIP, and a cache area is established on the server side. The cache is performed based on factors such as request frequency and data timeliness to reduce the occupancy of network transmission bandwidth and ensure that data can be transmitted to mobile terminals quickly and stably.

[0029] Secondly, the mobile user terminal module is equipped with an ARM Cortex-A78 processor, which provides sufficient graphics processing capabilities, 8GB of memory ensures smooth system operation, and a 2K screen resolution provides clear VR image display. The Bluetooth module supports Bluetooth 4.0 or higher standards for connecting external devices. Its software architecture includes browser components and device driver components. The browser component uses a Chromium-based browser kernel, which has the ability to parse and render VR content. When receiving game scene data sent by the cloud server, the built-in parser and renderer convert the data from the network transmission format to a screen-displayable format and render it, allowing players to see the dense jungle VR scene on the mobile phone screen. The browser kernel integrates an interface module for communicating with the tactile feedback device to receive and process information from the tactile feedback module. At the same time, A local cache mechanism is established, a cache area is opened in the local storage of the mobile phone, and the game scene data downloaded from the cloud server is cached. The cache update strategy is determined based on factors such as the timeliness of the content and the frequency of access. For example, for the frequently visited jungle area scene data, the cache is kept for a long time. When the player enters the area again, it can be read directly from the local cache, which speeds up the loading speed and reduces the amount of network data transmission. The device driver component develops corresponding device drivers for input devices such as handles. When the player operates the handle (connected via Bluetooth) to move, the device driver receives the input signal of the handle through the Bluetooth interface and converts it into instructions that the system can recognize, such as converting the joystick operation of the handle into corresponding movement instructions (forward, backward, left, right, etc.), and sends these instructions to the cloud server for processing.

[0030] Then, the hardware interface of the tactile feedback module provides a universal Bluetooth interface for the tactile feedback device on the mobile phone to connect, supports Bluetooth 4.0 or higher version standards, ensures the stability and speed of data transmission, and its software architecture includes a tactile feedback script parsing unit and a tactile effect generation unit, wherein the tactile feedback script parsing unit adopts a Python-based script parsing engine or a specially developed script parsing algorithm. When the player touches the virtual trees in the game, the script parsing engine receives the current game scene information and the pre-written tactile feedback script. The script adopts a JSON-like format, which contains the object information of the trees in the jungle scene, the touch action description and the corresponding tactile effect parameters (such as hardness value, roughness value, etc.). The parsing engine determines the specific parameters of the tactile effect to be generated according to the current scene information and the script content through lexical analysis and syntax analysis functions. For the vibration tactile feedback device, the tactile effect generation unit uses the formula Calculate the vibration intensity V p home, where is the vibration coefficient specific to the phone, which is related to hardware factors such as the performance of the vibration motor inside the phone and circuit design; H is the hardness value obtained by parsing the tactile feedback script, and H0 is the reference hardness value (which can be set to a hardness value representing air or a very soft object). is the touch type adjustment factor in the mobile phone application scenario (in the jungle game, the touch of the tree may be set to a specific value to distinguish the touch of different objects), C phone Indicates the current available power or computing resource ratio of the mobile phone (such as the remaining percentage of power or the ratio of CPU usage to total usage), C max Indicates the maximum value of the mobile phone power or computing resources, through the formula Calculate the vibration frequency f p home, where is a frequency coefficient specific to the mobile phone, which is related to the frequency response characteristics of the mobile phone vibration motor; R is the roughness value obtained by parsing the tactile feedback script, R0 is the reference roughness value (which can be set to represent the roughness value of a smooth surface), is the roughness type adjustment factor (used to distinguish the needs of different roughness simulations), S interaction Indicates the proportion of the touch-related interactive area on the current screen (when the player touches the area on the screen corresponding to the virtual tree, the proportion of this area to the total screen area), S total Represents the total screen area, based on the calculated vibration intensity V phone and the vibration frequency f phone , send control instructions to the vibration motor tactile feedback device through the Bluetooth interface, control the device to generate corresponding tactile sensations, simulate the hardness and roughness of trees, and when the player operates the virtual tool in the game, control the torque and displacement of the motor according to the parsed tactile effect parameters to generate corresponding force feedback effects and simulate the operating resistance of the tool, etc. For example, if the player uses a virtual axe to chop down trees, the corresponding tactile effect parameters are parsed according to the characteristics of the tool and the material characteristics of the tree, and then the torque and displacement of the force feedback device motor are controlled to make the player feel the resistance of the axe when chopping down trees. If the player's feedback tactile sensation is too strong, the tactile effect generation unit recalculates the parameters according to the feedback adjustment mechanism. For the vibration motor tactile feedback device, the vibration intensity and vibration frequency are recalculated to reduce the vibration intensity and frequency to make the tactile sensation more appropriate. For the force feedback device, the torque and displacement control parameters of the motor are readjusted to optimize the tactile effect.

[0031] Finally, the interaction module includes a hardware interface and a software architecture. The hardware interface provides a Bluetooth interface for connecting input devices such as handles, supports Bluetooth 4.0 or higher standards, and ensures the stability and speed of data transmission. The software architecture includes an input device interface component and a local interaction processing unit. The input device interface component develops a corresponding interface driver for the handle. When the player operates the handle to move, attack, etc., the interface driver receives the input signal of the handle through the Bluetooth interface, converts it into corresponding operation instructions (such as "forward", "backward", "attack", etc.), and sends these instructions to the cloud server for processing. The local interaction processing unit uses a rule engine to process simple interactive operations. When the player quickly switches the perspective, according to the angle change Δθ of the perspective switch and the interaction intensity factor I of the scene (in the adventure game scene, the I value is larger in a complex environment), the formula v=k is used. v ×Δθ×I to calculate the perspective switching speed v, where k v It is the speed coefficient, which is related to the performance of the mobile phone and the user settings. Then the viewing angle is adjusted directly to improve the interactive response speed. There is no need to send instructions to the cloud server module. When the player performs complex operations, such as using special skills, the local interactive processing unit sends the instructions to the cloud server module for processing.

[0032] Embodiment 2

[0033] This embodiment describes in detail a specific application of a cloud-based VR immersive experience system in the field of mobile education.

[0034] First, the cloud server module includes hardware architecture support and software architecture operation. For hardware architecture support, the cloud server adopts a high-performance server cluster, whose computing nodes are equipped with multi-core processors and large-capacity memory, and storage nodes adopt large-capacity disk arrays to provide powerful computing and storage capabilities for processing the scene data of the chemical virtual laboratory. For software architecture operation, it includes data storage layer, content processing layer and network communication layer. The data storage layer adopts Ceph distributed file system to store VR scene data of the chemical virtual laboratory (such as laboratory layout, experimental instrument model, reagent bottle model, etc.), object data (such as chemical reagents, experimental tools, etc.) and interactive scripts (such as experimental step scripts, instrument operation scripts, etc.). At the same time, MySQL or PostgreSQL database system is used to store metadata of these data, including scene name, applicable device type, creation time and other information. The distributed file system and database system are associated through the index mechanism to quickly locate and obtain the required The data and content processing layer integrates WebVR and WebXR technology related processing libraries, such as the relevant parts of the Three.js library. After receiving the performance parameters and application requests sent by the mobile phone, the acid-base neutralization experiment scene data suitable for the performance of the mobile phone is selected from the storage. Then, the scene data is converted into a format suitable for mobile browser rendering using the processing library, such as converting the 3D model into the WebGL format, and optimizing the texture and model to improve rendering efficiency and quality. The real-time rendering engine uses ray tracing related algorithms to simulate the scene lighting effects. For the light source on the laboratory table, its self-luminous intensity is set to a suitable value (representing the intensity of the light). Objects such as experimental instruments and reagent bottles have different reflection coefficients and transmission coefficients according to their own material characteristics. For example, a glass reagent bottle may have a higher transmission coefficient to simulate the effect of light penetration, and a metal instrument may have a higher reflection coefficient to simulate the reflection effect of its smooth surface. The illumination calculation formula L=L e +L r +L t Calculate the light intensity on the surface of each object, and the reflected light intensity L r , through the formula L r =k r ×L i ×F(θ i ,θ r ) calculation, where L i is the incident light intensity (calculated based on the distance from the light to the object and the intensity attenuation law), F(θ i ,θ r ) is the Fresnel reflection function, and the incident angle θ is determined by the angle between the light and the normal line of the object surface. i and the reflection angle θ r , for the transmitted light intensity L t , through the formula Lt =k t ×L i ×T(θ i ,θ t ) calculation, where T(θ i ,θ t ) is the transmission function, and the incident angle θ is also determined by the angle between the light and the normal line of the object surface. i and the transmission angle θ t At the same time, according to the scene object rendering sorting algorithm Sort the objects in the scene. In the chemical virtual laboratory scene, the value of α may be 0.6 (balancing distance and importance weight). For key objects such as instruments and reagent bottles on the laboratory table, their distance to the observation point (student perspective) is d i Relatively small, the importance weight w i Higher (because students pay more attention to these key items during the experiment), so the comprehensive ranking value S i Smaller ones are rendered first to improve rendering efficiency. The network communication layer uses HTTP / 2 or higher versions of the protocol to communicate with mobile user terminals, and uses the protocol's multiplexing and header compression features to improve data transmission efficiency. At the same time, the transmitted data is compressed with GZIP, and a cache area is established on the server side. The cache is performed based on factors such as request frequency and data timeliness to reduce the occupancy of network transmission bandwidth and ensure that data can be transmitted to mobile terminals quickly and stably.

[0035] Secondly, the mobile user terminal module is equipped with an ARM Cortex-A78 processor, which provides sufficient graphics processing capabilities, 8GB of memory ensures smooth system operation, and a 2K screen resolution provides clear VR image display. The Bluetooth module supports Bluetooth 4.0 or higher version standards for connecting external devices. Its software architecture includes browser components and device driver components. The browser component uses a Chromium-based browser kernel, which has the ability to parse and render VR content. When receiving virtual laboratory scene data sent by the cloud server, the built-in parser and renderer convert the data from the network transmission format to a screen displayable format and render it, allowing students to see the VR scene of the chemical experiment on the mobile phone screen. The browser kernel integrates an interface module for communicating with the tactile feedback device to receive and process information from the tactile feedback module. At the same time, a local cache is established. Mechanism, open a cache area in the local storage of the mobile phone, cache the virtual laboratory scene data downloaded from the cloud server, determine the cache update strategy according to factors such as the timeliness and access frequency of the content, for example, for the frequently accessed experimental step scene data, keep the cache for a long time, when the student enters the scene again, it can read directly from the local cache, speed up the loading speed, reduce the network data transmission volume, the device driver component develops a corresponding device driver for the glove input device, when the student operates the glove input device (connected via Bluetooth) to perform experimental operations, the device driver receives the sensor data on the glove through the Bluetooth interface, and converts it into instructions that the system can recognize, such as converting the sensor data of the bending of the glove finger into corresponding operation instructions (such as grabbing reagent bottles, operating instruments, etc.), and sends these instructions to the cloud server for processing.

[0036] Then, the hardware interface of the tactile feedback module provides a universal Bluetooth interface for the tactile feedback device on the mobile phone to connect, supports Bluetooth 4.0 or higher version standards, and ensures the stability and speed of data transmission. Its software architecture includes a tactile feedback script parsing unit and a tactile effect generation unit. The tactile feedback script parsing unit adopts a Python-based script parsing engine or a specially developed script parsing algorithm. When students touch the virtual reagent bottle in the virtual laboratory, the script parsing engine receives the current experimental scene information and the pre-written tactile feedback script. The script adopts a JSON-like format, which includes the object information of the reagent bottle in the chemical experiment scene, the touch action description and the corresponding tactile effect parameters (such as hardness value, roughness value, etc.). The parsing engine determines the specific parameters of the tactile effect to be generated according to the current scene information and the script content through lexical analysis and syntax analysis functions, and generates the tactile effect through the formula Calculate the vibration intensity V p hone, among which, is the vibration coefficient specific to the phone, which is related to hardware factors such as the performance of the vibration motor inside the phone and circuit design; H is the hardness value obtained by parsing the tactile feedback script, and H0 is the reference hardness value (which can be set to a hardness value representing air or a very soft object). is the touch type adjustment factor in the mobile phone application scenario (in the chemical experiment scenario, the touch of the reagent bottle may be set to a specific value to distinguish the touch of different objects), C phone Indicates the current available power or computing resource ratio of the mobile phone (such as the remaining percentage of power or the ratio of CPU usage to total usage), C max Indicates the maximum value of the mobile phone's heat or computing resources, through the formula Calculate the vibration frequency f p home, where is a frequency coefficient specific to the mobile phone, which is related to the frequency response characteristics of the mobile phone vibration motor; R is the roughness value obtained by parsing the tactile feedback script, R0 is the reference roughness value (which can be set to represent the roughness value of a smooth surface), is the roughness type adjustment factor (used to distinguish the needs of different roughness simulations), S interaction represents the proportion of the touch-related interactive area on the current screen (when the student touches the area on the screen corresponding to the virtual reagent bottle, the proportion of this area to the total screen area), S total It represents the total area of ​​the screen. According to the calculated vibration intensity and vibration frequency, a control instruction is sent to the vibration motor tactile feedback device through the Bluetooth interface to control the device to generate a corresponding tactile sensation and simulate the hardness and roughness of the reagent bottle. When students operate virtual experimental instruments in the virtual laboratory, such as using tweezers to pick up objects, the torque and displacement of the motor are controlled according to the parsed tactile effect parameters to generate a corresponding force feedback effect and simulate the operating resistance of the tweezers. For example, if students use virtual tweezers to pick up virtual chemical reagents, the corresponding tactile effect parameters are parsed according to the characteristics of the tweezers and the properties of the reagents, and then the torque and displacement of the motor of the force feedback device are controlled to make the students feel the resistance when the tweezers pick up the reagents. If the student feedback shows that the tactile sensation is too strong, the tactile effect generation unit recalculates the parameters according to the feedback adjustment mechanism. For the vibration motor tactile feedback device, the vibration intensity and vibration frequency are recalculated to reduce the vibration intensity and frequency to make the tactile sensation more suitable. For the force feedback device, the torque and displacement control parameters of the motor are readjusted to optimize the tactile effect.

[0037] Finally, the interaction module includes a hardware interface and a software architecture. The hardware interface provides a Bluetooth interface for connecting a glove input device and supports Bluetooth 4.0 or higher version standards to ensure the stability and speed of data transmission. The software architecture includes an input device interface component and a local interaction processing unit. The input device interface component develops a corresponding interface driver for the glove input device. When students operate the glove input device to perform experimental operations, the interface driver receives the sensor data on the glove through the Bluetooth interface, converts it into corresponding operation instructions (such as grabbing reagent bottles, operating instruments, etc.), and sends these instructions to the cloud server for processing. The local interaction processing unit uses a rule engine to process simple interactive operations. When students quickly switch perspectives, according to the angle change Δθ of the perspective switch and the interaction intensity factor I of the scene (in the chemical experiment scene, the I value is larger in complex environments), the formula v=k is used. v ×Δθ×I to calculate the perspective switching speed v, where k v It is the speed coefficient, which is related to the performance of the mobile phone and the user settings. Then the viewing angle is adjusted directly to improve the interactive response speed. There is no need to send instructions to the cloud server module. When students perform complex operations, such as precise operations to configure solutions, the local interactive processing unit sends the instructions to the cloud server module for processing.

[0038] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A cloud-based VR immersive experience system, characterized in that: The system includes the following components: cloud server module, user terminal module, tactile feedback module and interaction module; The cloud server module includes a hardware structure and a software structure, wherein the hardware structure includes a high-performance server cluster, wherein the server cluster includes multiple computing nodes and storage nodes, which are interconnected through a high-speed network, and the software structure includes a data storage layer, a content processing layer, and a network communication layer, wherein the data storage layer uses a distributed file system to store VR content data, and establishes a database system to store VR content metadata, the content processing layer has a real-time rendering engine, and can render VR content in real time according to user interaction operations, and the network communication layer uses HTTP / 2 and higher versions of the protocol to communicate with user terminals, and establish an efficient network connection management mechanism; The user terminal module can receive and parse VR content from the cloud server, and render and display it locally. The module also includes an interface connected to a tactile feedback device for receiving and processing tactile feedback information. The tactile feedback module is connected to the user terminal module and is used to interact with the advanced tactile feedback device. The module includes a tactile feedback script parsing unit and a tactile effect generating unit. The tactile feedback script parsing unit receives the current VR application scene information and the pre-written tactile feedback script. The parsing engine parses the script according to the grammatical structure of the script, extracts the scene object information, the touch action description and the corresponding tactile effect parameters, including hardness value, roughness value and temperature value, and determines the specific parameter value of the tactile effect to be generated according to the extracted information and the current application scene. The tactile effect generating unit generates the tactile effect parameters for the mobile phone tactile feedback device according to the tactile effect parameters determined by the tactile feedback script parsing unit by obtaining the vibration coefficient. Combining formulas Calculate the vibration intensity, where V phome Indicates the vibration intensity of the phone's tactile feedback device. is the vibration coefficient of the mobile phone, H is the hardness value obtained by analyzing the tactile feedback script, H0 is the reference hardness value, It is the touch type adjustment factor in the mobile phone application scenario. is the correlation coefficient of the current state of the mobile phone, C phone Indicates the current available power or computing resource ratio of the mobile phone, C max Indicates the maximum value of the mobile phone power or computing resources, and according to the formula Calculate the vibration frequency, where f phome Indicates the vibration frequency of the phone's tactile feedback device. is the frequency coefficient specific to the mobile phone, R is the roughness value obtained by parsing the tactile feedback script, and R0 is the base roughness value. is the roughness type adjustment factor, is the coefficient related to the mobile phone screen interaction, S interaction Indicates the proportion of the touch-related interactive area on the current screen, S total Indicates the total area of ​​the screen, and is used to dynamically adjust the vibration frequency according to the user's interactive position and area on the screen, to provide tactile feedback that is more in line with the user's operation. According to the calculated vibration intensity and vibration frequency, a control instruction is sent to the vibration motor tactile feedback device through the Bluetooth interface to control the device to generate a corresponding tactile sensation. During the tactile feedback process, the user's feedback information is continuously monitored. If the user feedback is that the tactile sensation is too strong or too weak, the tactile effect parameters are adjusted according to the user feedback. For the vibration motor tactile feedback device, the vibration intensity and vibration frequency are recalculated to optimize the tactile effect; The interaction module is used to realize real-time interaction between the user and the VR virtual environment. At the same time, the interaction module also includes a local interaction processing unit on the user terminal device, which is used to process the interaction logic.

2. A cloud-based VR immersive experience system according to claim 1, characterized in that: The cloud server module includes a real-time rendering engine that can render VR content in real time according to the user's interactive operation. The rendering order of objects is determined by comprehensively considering the distance from the object to the observation point and the importance weight of the object. Specifically, for an object in a three-dimensional space, its position coordinate is (x i ,y i ,z i ), the observation point coordinates are (x0, y0, z0), then the distance Determine the importance weight w of the object based on scene design and interaction requirements i , calculate the comprehensive ranking value Among them, α is the weight coefficient. By adjusting the value of α, the influence of distance and importance weight on the ranking can be balanced.

3. The cloud-based VR immersive experience system according to claim 1, characterized in that: The cloud server module includes a real-time rendering engine that can render VR content in real time according to the user's interactive operation. Specifically, for each object to be rendered, the reflection coefficient k in its material information is obtained according to the order determined by the rendering sorting algorithm. r and the transmission coefficient k t parameters, and determine the incident light intensity L i , calculate the incident light intensity L according to the intensity of the light source in the illumination information table and the relative position and direction relationship between the object and the light source i , let the light source intensity be I0, and the distance from the light source to the object be d0, then the incident light intensity In addition, the reflected light intensity L r The calculation formula is: L r =k r ×L i ×F(θ i ,θ r ), where θ i is the angle of incidence, θ r is the reflection angle, F(θ i ,θ r ) is the Fresnel reflection function, and the incident angle and reflection angle are determined by the angle between the light and the normal line of the object surface, and the transmitted light intensity L is calculated t =k t ×L i ×T(θ i ,θ t ), where θ i is the angle of incidence, θ t is the transmission angle, T(θ i ,θ t ) is the transmission function. Similarly, the incident angle and transmission angle are determined by the relationship between the light and the surface normal of the object, and the final light intensity L = L is calculated. e +L r +L t , where L e is the self-luminous intensity of the object. For objects without self-luminous properties, L e =0.

4. The cloud-based VR immersive experience system according to claim 1, characterized in that: The distributed file system of the data storage layer of the cloud server module adopts Ceph, the database system adopts MySQL and PostgreSQL, and the WebVR and WebXR technology related processing library of the content processing layer is used to convert the format and optimize the selected VR content to make it suitable for user terminal devices.

5. The cloud-based VR immersive experience system according to claim 1, characterized in that: The user terminal module includes hardware support and software architecture. Its hardware support includes that the P device requires a memory of more than 16GB, the network interface is a Gigabit Ethernet interface or a high-speed Wi-Fi module, and the Android device and tablet require the processor to be an ARM Cortex-A series high-end processor, with a memory of more than 8GB and a screen resolution of 2K or higher.

6. The cloud-based VR immersive experience system according to claim 5, characterized in that: The user terminal module includes hardware support and software architecture, and its software architecture includes a browser component and a device driver component. The browser component adopts a browser kernel that supports WebVR and WebXR technologies. The kernel integrates an interface module for communicating with a tactile feedback device and establishes a local cache mechanism. The device driver component develops corresponding drivers for different input devices and tactile feedback devices.

7. The cloud-based VR immersive experience system according to claim 1, characterized in that: The interaction module includes a hardware interface and a software architecture, wherein the hardware interface provides multiple interfaces to connect input devices, including a USB interface, a Bluetooth interface and a sensor interface.

8. The cloud-based VR immersive experience system according to claim 7, characterized in that: The software architecture of the interactive module includes an input device interface component and a local interactive processing unit. The input device interface component develops corresponding interface drivers for different input devices. The local interactive processing unit uses a rule engine to process simple interactive operations and maintains communication with the cloud server module. When encountering complex interactive operations, it sends instructions to the cloud server module.

Images