Content dynamic generation and immersive sightseeing interaction system

Abstract

The invention relates to the technical field of interaction, and discloses a content dynamic generation and immersive sightseeing interaction system, which comprises a user behavior capture module used for capturing the position, sight direction and interaction action of a user in a virtual environment in real time and generating user behavior data; the distributed optimal control module is connected with the user behavior capturing module and is used for generating a global control strategy based on the user behavior data generation content; and the content generation module is connected with the distributed optimal control module. According to the invention, a technology of combining a distributed optimal control module and user behavior modeling is adopted, and the technical effect of highly matching dynamic content generation and user behaviors is achieved. Compared with a scheme of generating content depending on a fixed rule in the prior art, the method solves the defect that content generation cannot adapt to dynamic behaviors of the user in real time, and greatly improves the immersion and the degree of freedom of the user in the virtual scene.

Description

Technical Field

[0001] The present invention relates to the field of interaction technologies, and specifically to a content dynamic generation and immersive tour interaction system. Background Art

[0002] In recent years, virtual reality technology has been widely used in fields such as education, culture and tourism, and industrial simulation. Through immersive interaction experiences, VR technology enables users to freely explore and operate in virtual environments, providing a deep sense of immersion that traditional display technologies cannot achieve. Especially in the field of dynamic content generation and interaction in virtual scenarios, the development of technology has gradually brought the user experience closer to realism and immediacy.

[0003] In the prior art, dynamic content generation usually relies on preset fixed rules or simple real-time generation algorithms to achieve. Although these technical solutions can meet the basic experience needs of users to a certain extent, in complex scenarios, such as large-scale immersive environments with multiple users participating, existing systems are difficult to respond to user behaviors in real time and generate virtual content that precisely matches. In addition, the combination of physical interaction and multi-modal feedback is less, and most technical solutions only support single-sensory feedback, making it difficult for the user experience to achieve a truly immersive effect.

[0004] The main problem existing in the prior art is the insufficient matching degree between dynamic content generation and user behaviors. In complex scenarios, content generation fails to effectively adapt to the real-time position, actions, and line-of-sight directions of users, resulting in inconsistent virtual content and user behaviors, seriously affecting the user's sense of immersion and interaction experience. This problem limits the development needs of virtual reality technology in higher-level application scenarios. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides a content dynamic generation and immersive tour interaction system to solve the problem in the prior art that dynamic content generation cannot adapt to user behaviors in real time, resulting in a mismatch between virtual content and user interaction requirements.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: A content dynamic generation and immersive tour interaction system, including: A user behavior capture module, configured to capture the position, line-of-sight direction, and interaction actions of a user in a virtual environment in real time, and generate user behavior data; A distributed optimal control module, connected to the user behavior capture module, configured to generate a global control strategy for content generation based on the user behavior data; A content generation module, connected to the distributed optimal control module, configured to dynamically generate virtual content that matches the user behavior according to the global control strategy; A physical simulation module, connected to the content generation module, is used to calculate the interaction response based on the physical mechanics model of the virtual content and user interaction process; A multimodal feedback module, connected to the physical simulation module, is used to provide visual, tactile, and audio feedback to the user according to the interaction response to enhance the user's immersion.

[0007] Preferably, the user behavior capture module includes: A spatial positioning unit, used to capture the real-time position data of the user; An action recognition unit, used to recognize the user's controller operations and gesture actions, and the behavior data generated by the spatial positioning unit and the action recognition unit is transmitted to the distributed optimal control module.

[0008] Preferably, the distributed optimal control module includes: A user behavior modeling unit, connected to the user behavior capture module, is used to establish a dynamic behavior distribution based on the user's position and interaction behavior; A control strategy optimization unit, connected to the user behavior modeling unit, is used to calculate the optimal control strategy for virtual content generation based on the dynamic behavior distribution.

[0009] Preferably, the user behavior modeling unit describes the user behavior range through a probability distribution function and transmits the modeling result to the control strategy optimization unit to generate a global control strategy.

[0010] Preferably, the content generation module includes: A dynamic content hierarchical loading unit, connected to the distributed optimal control module, is used to preferentially generate high-precision content according to the user's line of sight range and dynamically load the content outside the line of sight range with low precision; A sparse optimization generation unit, connected to the dynamic content hierarchical loading unit, is used to dynamically generate virtual content through a sparse regularization generation network.

[0011] Preferably, the sparse optimization generation unit receives the generation priority instruction of the dynamic content hierarchical loading unit and reduces unnecessary content generation calculations through sparse regularization constraints.

[0012] Preferably, the physical simulation module includes: An interaction mechanics modeling unit, connected to the content generation module, is used to construct a multi-body interaction model based on Lagrangian mechanics according to the physical properties of the virtual content and the user interaction actions; An interaction response calculation unit, connected to the interaction mechanics modeling unit, is used to calculate the dynamic response of the virtual content according to the multi-body interaction model.

[0013] Preferably, the interaction response calculation unit determines the motion trajectory of the virtual content under the user's force by calculating the changes in kinetic energy and potential energy of the virtual content, and transmits the calculation result to the multimodal feedback module.

[0014] Preferably, the multimodal feedback module includes: A visual feedback unit, connected to the physical simulation module, for adjusting the user's visual experience according to the light and shadow changes of the virtual content; A tactile feedback unit, connected to the physical simulation module, for providing an interactive force feeling to the user through a vibration module; An audio feedback unit, connected to the physical simulation module, for dynamically adjusting the sound effect based on the user's interaction position and actions.

[0015] Preferably, it further includes a closed-loop interaction control module, connected to the user behavior capture module and the distributed optimal control module, for dynamically adjusting the global control strategy of content generation based on the user's real-time feedback.

[0016] The present invention provides a content dynamic generation and immersive tour interaction system. It has the following beneficial effects: 1. The present invention adopts the combined technology of a distributed optimal control module and user behavior modeling, achieving the technical effect of highly matching dynamic content generation with user behavior. Compared with the prior art solutions that rely on fixed rules to generate content, it solves the problem that content generation cannot adapt to user dynamic behavior in real time, significantly enhancing the immersion and freedom of the user in the virtual scene.

[0017] 2. The present invention improves the content generation efficiency through a sparse optimization generative adversarial network, achieving the technical effects of preferentially loading high-resolution content and reasonable resource allocation. Different from the high computational load brought by full-scene high-resolution rendering in the prior art, sparse optimization reduces unnecessary waste of computing resources, solves the problem of low system operation efficiency, and at the same time ensures the visual quality of the core interaction area.

[0018] 3. The present invention uses the Lagrangian mechanical model for physical simulation calculation, which can accurately simulate the interactive mechanical relationship between the user and the virtual content, achieving the technical effect of high-fidelity interaction response. Compared with the traditional virtual interaction method based on simple rigid body collision calculation, the present invention solves the problems of untrue interactive physical response and lack of details in the prior art, significantly enhancing the dynamic expressiveness of the virtual content.

[0019] 4. The present invention integrates visual, tactile, and audio signals through a multimodal feedback module, providing an all-round immersive interaction experience and achieving the technical effects of real-time and multi-sensory consistency. In the prior art, there is usually a single reliance on visual or tactile feedback, which is difficult to cover the complete interaction perception. However, the present invention effectively solves the problems of fragmented and unnatural user interaction experiences, enabling users to truly feel the dynamic changes of virtual content. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is the overall system architecture flowchart of the present invention; Figure 2 is the flowchart of the user behavior capture module of the present invention; Figure 3 is the flowchart of the content generation module of the present invention; Figure 4 is the flowchart of the physical simulation and feedback module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0022] Please refer to the attached Figures 1-4 , the embodiment of the present invention provides a content dynamic generation and immersive tour interaction system, including.

[0023] 1. User behavior capture module The user behavior capture module continuously obtains the position, line-of-sight direction, and interaction actions of the user in the virtual environment, providing key data input for the subsequent distributed optimal control module. Through high-precision spatial positioning and action recognition, this module constructs a dynamic distribution model of user behavior, thereby ensuring a close association between user behavior and virtual content generation. Generally, this module maintains a real-time connection with the subsequent modules through a data flow interface, and the captured behavior data is directly transmitted to the distributed optimal control module for further processing. The user behavior capture module includes a spatial positioning unit and an action recognition unit.

[0024] Spatial positioning unit Specifically, the spatial positioning unit uses laser positioning or optical tracking technology to achieve real-time positioning of the user in the virtual environment. As a possible implementation, the spatial positioning unit determines the three-dimensional coordinates of the user through the optical reflection signals between the head-mounted device and the external positioning base station. 。This coordinate data is used to dynamically describe the user's current location. In a possible implementation, the user's current location is modeled in the form of a probability distribution, which is used to represent the range and dynamic change trend of the user's behavior. The distribution function can be expressed in the following form:

[0025] Where: Represents the probability density of the user's behavior at location and time ; Is the user's current location, which depends on the real-time measurement results of the spatial positioning unit; Is the standard deviation, which reflects the activity range of the user's behavior.

[0026] As an option, the system can dynamically adjust the parameter .

[0027] For example, when the user enters a narrow area of the exhibition hall, The value can be reduced to improve the resolution of the model.

[0028] On the contrary, when the user is in an open area, The value can be appropriately increased to describe a wider range of behaviors. In some embodiments, the spatial positioning unit improves the positioning accuracy through multi-sensor fusion technology. Specifically, the inertial measurement unit is combined with the optical tracking technology, and the data of the IMU is used as a supplement to the optical positioning to maintain high positioning stability under occlusion.

[0029] Action recognition unit In this embodiment, the action recognition unit is used to capture the user's gesture actions, controller operations or other interaction behaviors. In a possible implementation, the action recognition unit captures the user's grasping, rotating, releasing and other actions based on the attitude sensors and button trigger signals built into the controller. For example, when the user presses the grasping button on the controller, the system will recognize that the user is trying to interact with Virtual objects. Specifically, the action recognition unit adopts a set of behavior parsing algorithms based on sensor data, including attitude solution and action trigger determination. Generally, the attitude solution is carried out through the following formula:

[0030] Where: Represents the current attitude of the controller, Obtained by integrating the angular velocity measured by the gyroscope; Obtained by correcting the gravity direction data of the acceleration sensor. As an option, the action recognition unit can combine machine learning algorithms to improve the recognition accuracy of complex gestures.

[0031] Specifically, train an action classification model based on the user's hand data, input the acceleration and angular velocity data collected by the sensor into the neural network, and output the action categories that the user may perform.

[0032] Data flow and connection In the present invention, the user behavior capture module is connected to the distributed optimal control module in real time. User behavior data, including the three-dimensional coordinate data of the spatial positioning unit and the behavior category data of the action recognition unit, will be transmitted to the control module in real time. The distributed optimal control module uses this data to establish a dynamic distribution model of user behavior to guide virtual content generation.

[0033] Specifically, the coordinate data generated by the spatial positioning unit will be directly passed to the user behavior modeling unit as the core input parameter of the dynamic distribution model. The action type data generated by the action recognition unit will be used as an auxiliary parameter of the distribution to adjust the type and interaction method of the generated content. For example, the recognition result of the user's grasping action will guide the content generation module to generate an interactive object.

[0034] The user behavior capture module can also be extended to a multi-user collaboration scenario. In this case, the system synchronously captures the behavior data of multiple users through multiple spatial positioning units and action recognition units. The behavior data of each user is independently modeled to form a multi-distribution model for describing the behavior dynamics of multiple users. Specifically, the user behavior distribution can be extended to the following form:

[0035] Where: Represents the number of currently participating users; Is the behavior weight of the th user, usually related to the interaction frequency of the user; And Respectively represent the current position and behavior range of the th user. As a possible implementation, in a multi-user scenario, the system can preferentially generate high-priority content for users with a higher interaction frequency.

[0036] 2. Distributed Optimal Control Module The distributed optimal control module realizes the dynamic mapping between user behavior data and content generation strategies. The distributed optimal control module receives the data output by the user behavior capture module, establishes a dynamic model of user behavior, combines the global state of the system, and calculates the optimal control strategy for virtual content generation. Generally, this module processes complex calculation tasks through a distributed optimization algorithm, controls the response time of the overall system within the millisecond level, and at the same time ensures that the generated content highly matches the user behavior.

[0037] In this embodiment, the distributed optimal control module includes a user behavior modeling unit and a control strategy optimization unit.

[0038] User behavior modeling unit In this embodiment, the user behavior modeling unit provides basic data support for subsequent control strategy generation by dynamically modeling user behavior data. As a possible implementation, the user behavior modeling unit describes the dynamic distribution of user behavior based on the Gaussian distribution model. The model uses the captured user location and time data to generate the behavior probability distribution of the user in the virtual environment . Generally, this distribution is expressed by the following formula:

[0039] Where: represents the behavior probability of the user at location and time ; is the real-time location of the user, provided by the user behavior capture module; is the standard deviation of the behavior range, indicating the size of the area where the user may move.

[0040] Specifically, the value of can be adjusted according to the characteristics of the virtual environment. In a relatively narrow area, such as the aisle in a virtual museum, can be reduced to more accurately reflect the behavior range of the user. In a relatively open area, such as the center of the exhibition hall, can be appropriately increased to cover more possible movement directions of the user.

[0041] In some embodiments, the user behavior modeling unit can also be extended to a multi-user scenario. At this time, the behavior distribution model is extended to a weighted superposition form of multi-user behavior distribution:

[0042] Where: Represents the number of users; is the behavior weight of the th user and is related to the interaction frequency of the user; and respectively represent the current position and behavior range of the th user. In a possible implementation, the user behavior modeling unit optimizes the real-time update of the multi-user behavior distribution through multi-threaded parallel computing. This method improves the modeling efficiency of the system for multi-user behavior.

[0043] Control strategy optimization unit In this embodiment, the control strategy optimization unit calculates the global optimal control strategy for virtual content generation based on the user behavior distribution . As an option, the control strategy optimization unit adopts the distributed optimal control theory and improves the optimization efficiency by decomposing the global problem into multiple sub-problems for parallel solution. Specifically, this unit first constructs a global cost function to describe the matching degree between user behavior and generated content. The form of the cost function is as follows:

[0044] Where: is the control strategy, that is, the parameter used to guide content generation; is the deviation degree between user behavior and generated content, usually described by the difference between the user behavior distribution and the content distribution ; is the energy consumption term of the control strategy, used to limit the complexity of the control strategy.

[0045] In a possible implementation, the control strategy optimization unit distributes and solves the optimal solution of the cost function through the alternating direction method of multipliers (ADMM).

[0046] The specific steps are as follows: First, decompose the global cost function into multiple sub-cost functions, and each sub-function corresponds to a user behavior node; Then optimize its control strategy in parallel at each node; Finally, through the global coordination mechanism, merge the optimization results of each node to generate the global control strategy. In some embodiments, in order to improve real-time performance, the control strategy optimization unit adopts the dynamic programming method to further reduce the computational complexity. Dynamic programming decomposes the global problem into multiple time-series sub-problems for step-by-step optimization through phased solution.

[0047] Data connection and module interaction Under normal circumstances, the distributed optimal control module maintains a data stream connection with the user behavior capture module. User behavior data and action feature information are used as inputs to the modeling unit, and the generated user behavior distribution is directly passed to the control strategy optimization unit.

[0048] The control strategy generated by the control strategy optimization unit is transmitted to the content generation module in real time. The control strategy determines the generation location, resolution, and priority of the virtual content. For example, when the user is in a high-priority area, the strategy will guide the content generation module to preferentially load high-resolution content.

[0049] As an option, the distributed optimal control module can also interact with the multimodal feedback module. The updated results of the control strategy can adjust the parameters of visual, tactile, and audio feedback, forming a closed-loop control of user behavior and content generation.

[0050] The distributed optimal control module can combine machine learning methods to further improve the adaptability of the control strategy. For example, a policy generation network is trained using a reinforcement learning algorithm, and the weight parameters in the cost function are dynamically adjusted through long-term observation of user behavior to adapt to the personalized needs of users.

[0051] The distributed optimal control module supports the adjustment of dynamic environments. The module can sense changes in the virtual environment, such as changes in the layout of exhibits or scene switching, and recalculate the user behavior distribution and control strategy to ensure that the generated content is consistent with the environmental changes.

[0052] 3. Content Generation Module The content generation module is responsible for dynamically generating virtual content that matches the user behavior according to the control strategy output by the distributed optimal control module. Under normal circumstances, the content generation module receives the guidance information of the control strategy in real time, combines the current position and line-of-sight range of the user, preferentially generates virtual content in high-priority areas, and dynamically adjusts the content loading method in low-priority areas. The goal of this module is to reduce the waste of system computing resources and improve the response speed of the virtual scene through an efficient content generation strategy.

[0053] In this embodiment, the content generation module includes a dynamic content hierarchical loading unit and a sparse optimization generation unit.

[0054] Dynamic Content Hierarchical Loading Unit In this embodiment, the dynamic content hierarchical loading unit divides the priorities of content generation according to the user's line of sight range, and adopts a hierarchical loading method to optimize resource allocation. Specifically, the dynamic content hierarchical loading unit takes the user's line of sight direction as the center and delimits a high-priority area, and the content within this area is generated at a high resolution. At the same time, the area outside the line of sight is delimited as a low-priority area, and the content in the low-priority area is loaded at a low resolution or loaded later. Generally, the size of the line of sight range can be dynamically adjusted according to the scenario requirements. In one possible implementation, the user's line of sight range can be described by the following function:

[0055] Where: represents the priority of the content at the position; is the center point position of the user's line of sight; is the radius of the line of sight range, representing the coverage range of the high-priority area.

[0056] As an option, the dynamic content hierarchical loading unit can dynamically adjust the high-priority area in combination with the results of user behavior modeling. For example, when the user moves quickly in the scene, the range of the high-priority area is appropriately increased to avoid loading delays affecting the user experience. In some embodiments, the dynamic content hierarchical loading unit also supports a multi-user collaboration scenario. In this case, the module superimposes the line of sight ranges of multiple users to generate a joint priority distribution for guiding content loading. The joint priority distribution can be calculated by the following formula:

[0057] Where: is the joint priority distribution; is the current number of users; is the th user's priority distribution; Sparse optimization generation unit In this embodiment, the sparse optimization generation unit uses a sparse regularized generative adversarial network (Sparse-GAN) to dynamically generate virtual content. The goal of this unit is to reduce unnecessary content generation calculations and improve generation efficiency through sparse optimization. As a possible implementation, the sparse optimization generation unit adds a sparsity constraint to the generator objective function to achieve efficient content generation. The generator's objective function can be expressed as:

[0058] Among them: is the generator network, used to generate virtual content; is the discriminator network, used to evaluate the authenticity of the generated content is a random noise vector, used as the input for generating content; is the weight of the sparsity constraint; is the sparse regularization term, used to control the sparsity of the generated content.

[0059] Specifically, the sparse optimization generation unit can preferentially generate high-priority content within the line of sight, while reducing the generation complexity of low-priority areas. In some embodiments, the sparse optimization generation unit adopts a step-by-step generation strategy, that is, generating virtual content in order of priority from high to low. This strategy ensures the real-time nature of the generation process and avoids delays caused by over-allocation of resources.

[0060] As an option, the sparse optimization generation unit can also combine the results of the dynamic content hierarchical loading unit to generate a content cache for high-priority areas in advance, so as to further improve the response speed. For example, when the user quickly moves to a new area, the system can directly load the generated cached content without recalculation.

[0061] Data connection and module interaction The content generation module directly receives the control strategy output by the distributed optimal control module. The control strategy includes the generation location, resolution, and priority information of the content. These information are parsed by the dynamic content hierarchical loading unit and used to guide the hierarchical loading of content generation.

[0062] The sparse optimization generation unit adjusts the generation parameters of the generator according to the priority information provided by the dynamic content hierarchical loading unit. For example, when the priority of a certain area is low, the generator can reduce the resolution of that area to reduce the consumption of computing resources.

[0063] The generated virtual content will be passed to the physical simulation module for interactive calculation between the user and the content. At the same time, the generated content will also be fed back to the multi-modal feedback module to provide an immersive experience of vision, touch, and audio.

[0064] In some embodiments, the content generation module can combine reinforcement learning techniques to optimize the generation strategy. By observing the user's behavior in the long term, the system can dynamically adjust the weight parameters of the generator, thereby improving the matching degree between the generated content and the user's needs.

[0065] In another possible implementation, the content generation module supports dynamic scene switching. When the user switches from one virtual scene to another, the system can quickly generate new scene content and release the memory of the old scene content to ensure the operation efficiency of the system.

[0066] Physical simulation module In the content dynamic generation and immersive tour interaction system of the present invention, the physical simulation module undertakes the dynamic calculation task of the interaction behavior between the user and the virtual content. Based on the virtual content output by the content generation module and the user behavior data, this module uses the simulation technology based on the Lagrangian mechanics model to calculate the physical response state of the virtual content in real time. Generally, the interaction results generated by the physical simulation module are directly transmitted to the multi-modal feedback module to provide an immersive user interaction experience.

[0067] In this embodiment, the physical simulation module mainly includes an interaction mechanics modeling unit and an interaction response calculation unit.

[0068] Interaction mechanics modeling unit In this embodiment, the interaction mechanics modeling unit constructs the interaction mechanics relationship between the user and the virtual content based on the Lagrangian mechanics model. The Lagrangian mechanics model can simultaneously describe the motion behavior and interaction state of the content.

[0069] Specifically, the motion of the virtual content is determined by the changes in its kinetic energy and potential energy. The Lagrangian function can be expressed by the following formula:

[0070] Where: represents the Lagrangian of the system; represents the kinetic energy of the virtual content; represents the potential energy, including the internal potential energy of the interaction between the contents and the external potential energy caused by the external environment is the mass of the virtual content; is the displacement vector of the virtual content; is its velocity vector. Generally, the internal potential energy is used to describe the elastic connection or collision effect between the virtual contents, and the external potential energy is used to simulate environmental factors such as gravity and friction.

[0071] When a user interacts with virtual content, the interaction force introduces additional potential energy terms. For example, when the user applies a grasping action through a controller, the total potential energy of the virtual content can be expressed as:

[0072] where represents the additional potential energy introduced by user interaction, which is usually proportional to the force applied by the user. As an option, the interaction mechanics modeling unit can also dynamically adjust the model parameters to adapt to different content types. For example, for virtual content with a larger mass, the stiffness coefficient of the internal potential energy can be increased to avoid excessive deformation of the content during interaction.

[0073] Interaction response calculation unit In this embodiment, the interaction response calculation unit calculates the motion trajectory and state changes of the virtual content in real time according to the Lagrangian function output by the interaction mechanics modeling unit. The calculation process is based on the Euler - Lagrange equation, and the specific formula is as follows:

[0074] where: represents the external force acting on the virtual content, including user interaction force, gravity, etc.

[0075] The interaction response calculation unit uses numerical integration methods to solve the Euler - Lagrange equation. Commonly used numerical integration methods include the Runge - Kutta method and the Verlet integration method. Through these methods, the motion state of the virtual content under the user's applied force, including displacement, velocity, and acceleration, can be calculated quickly.

[0076] To improve the simulation efficiency, the interaction response calculation unit partitions the virtual content in complex scenes. Specifically, the scene is divided into multiple independent interaction regions, and the content interaction in each region is calculated by an independent physics engine. The data between regions is synchronized through boundary conditions to ensure the consistency of the global physical response.

[0077] The interaction response calculation unit can combine the dynamic data of the user's behavior to optimize the physical properties of the virtual content. For example, when the user quickly grasps a certain virtual object, the module can temporarily reduce the mass parameter of the object , to improve the response speed of the content.

[0078] Data connection and module interaction Under normal circumstances, the physical simulation module and the content generation module maintain a real-time connection through a data interface. The virtual content attribute data provided by the content generation module, including mass, initial position, and geometry, is directly transmitted to the interactive mechanics modeling unit for constructing a mechanics model. At the same time, the interactive force data generated by the user behavior capture module is also transmitted to the interactive response calculation unit for calculating the force exerted by the user on the virtual content. For example, when the user presses a controller button, the signal is parsed into an external force , and is superimposed on the external force term . The simulation results, such as the real-time displacement and velocity data of the virtual content, are transmitted to the multi-modal feedback module for providing dynamic feedback in vision, touch, and audio.

[0079] The physical simulation module can incorporate particle system technology to simulate complex effects such as liquids and smoke. Specifically, the virtual content is divided into multiple particles, and the motion trajectory of each particle is calculated through the Lagrangian mechanics model. The mutual forces between particles can be described by a potential energy function . In another possible implementation, the physical simulation module supports hybrid simulation of rigid bodies and soft bodies. For example, a rigid body can be used to simulate a tool grasped by the user, and a soft body can be used to simulate an elastic object grasped by the user. The module integrates the motion calculations of rigid bodies and soft bodies through a unified mechanics modeling framework.

[0080] 5. Multi-modal feedback module The multi-modal feedback module is an output module directly facing the user, responsible for feedbacking the interaction information between the user and the virtual content to the user in various forms of vision, touch, and audio.

[0081] This module receives the real-time interaction data output by the physical simulation module and converts it into a feedback signal that conforms to the sensory experience. Generally, the design goal of the multi-modal feedback module is to provide natural, real-time, and highly immersive interaction feedback to enhance the authenticity of the virtual scene.

[0082] In this embodiment, the multi-modal feedback module includes a visual feedback unit, a tactile feedback unit, and an audio feedback unit Visual feedback unit In this embodiment, the visual feedback unit provides real-time visual feedback to the user by dynamically adjusting the lighting effects, material properties, and dynamic changes of the virtual content. Specifically, the visual feedback unit uses ray tracing technology to simulate the lighting changes in the scene. The material properties of the virtual content, such as smoothness, reflectivity, and transparency, are adjusted in real time according to the content state output by the physical simulation module. For example, when the user drags a smooth metal ball, the intensity of the light reflected on its surface changes dynamically with the moving direction of the user. These lighting changes are calculated by the following formula:

[0083] Wherein: is the light intensity currently observed by the user is the initial light intensity of the light source; is the angle between the user's line of sight direction and the light reflection direction.

[0084] As an option, the visual feedback unit can also use a particle system to simulate dynamic effects, such as explosions, smoke, or fluid motion. The generation rate, velocity, and color of the particles can be dynamically adjusted according to the user's interaction intensity.

[0085] The visual feedback unit optimizes the display effect through dynamic resolution adjustment technology. Specifically, the content within the user's line of sight is rendered at a high resolution, while the content outside the line of sight is rendered at a reduced resolution to reduce the rendering calculation amount.

[0086] Tactile feedback unit In this embodiment, the tactile feedback unit uses a vibration module on the controller or other tactile devices to feedback the interaction force between the user and the virtual content to the user in the form of vibration signals. Generally, the intensity and frequency of the tactile feedback signals are proportional to the interaction force applied by the user. For example, when the user grabs a virtual object, the tactile feedback unit generates the following vibration signals according to the mechanical data calculated by the physical simulation module:

[0087] Wherein: is the feedback intensity of the tactile vibration, is the interaction force applied by the user; is the feedback coefficient, used to control the sensitivity of the vibration signal.

[0088] As a possible implementation, the tactile feedback unit supports different types of tactile modes, such as continuous vibration mode and pulse vibration mode. The continuous vibration mode is used to simulate a constant interaction force, such as the resistance feeling when the user holds an object; the pulse vibration mode is used to simulate an instantaneous interaction force, such as the force feedback when the user quickly clicks an object.

[0089] The tactile feedback unit can also adjust the frequency of the vibration signal according to the material properties of the virtual content. For example, when the user touches a rough surface, the frequency of the tactile feedback is lower; when the user touches a smooth surface, the frequency of the tactile feedback is higher.

[0090] Audio feedback unit In this embodiment, the audio feedback unit provides audio feedback with a strong sense of space through dynamic sound field technology. The generation of the audio feedback signal is based on the interaction position, actions, and content characteristics between the user and the virtual content. Specifically, the audio feedback unit adjusts the volume according to the distance between the user and the virtual content. The volume change can be described by the following formula

[0091] where: is the volume currently heard by the user; is the initial volume; is the distance between the user and the virtual content.

[0092] As an option, the audio feedback unit can also dynamically adjust the directionality of the audio. When the user approaches a sound-emitting object, the direction of the audio signal is adjusted according to the user's auditory localization. For example, when the user stands on the left side of a virtual fountain, the sound effect of the fountain will be emitted from the direction of the user's left ear. In some embodiments, the audio feedback unit supports ambient sound simulation, such as wind sound, rain sound, or background music. The volume and frequency of these ambient sounds can be adjusted in real time according to the changes in the virtual scene.

[0093] Data connection and module interaction The multimodal feedback module is closely connected to the physical simulation module and receives the simulation calculation results in real time. For example, the visual feedback unit adjusts the display state of the virtual content according to the content displacement and rotation data output by the physical simulation module; the tactile feedback unit generates corresponding vibration signals according to the interaction force applied by the user; the audio feedback unit generates spatial sound effects according to the position and state of the virtual content.

[0094] The multimodal feedback module also interacts with the user behavior capture module. When the user changes the position or the direction of the line of sight, the feedback signal is dynamically updated according to the new user state to ensure the real-time and consistency of the feedback effect.

[0095] The multimodal feedback module can combine artificial intelligence technology to predict the feedback needs of the user. For example, by analyzing the interaction patterns of the user, the system can generate corresponding feedback signals in advance, thereby further reducing the feedback delay.

[0096] In another possible implementation, the multimodal feedback module supports custom feedback settings. The user can adjust the intensity, frequency, or type of the feedback signal according to personal preferences to obtain an interaction experience that better meets the personalized needs.

[0097] Although embodiments of the present invention have been shown and described, those of ordinary skill in the art will appreciate that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Dynamic content generation and immersive tour interactive system, characterized by: include: User behavior capture module, used to capture the user's position, sight direction and interactive actions in the virtual environment in real time and generate user behavior data; A distributed optimal control module, connected to the user behavior capture module, for generating a global control strategy for content generation based on the user behavior data; A content generation module, connected to the distributed optimal control module, for dynamically generating virtual content matching user behavior according to the global control strategy; A physical simulation module, connected to the content generation module, for calculating the interaction response based on the physical mechanics model of the interaction process between the virtual content and the user; The multimodal feedback module is connected to the physical simulation module and is used to provide visual, tactile and audio feedback to the user according to the interactive response to enhance the user's sense of immersion.

2. The content dynamic generation and immersive tour interactive system according to claim 1 is characterized in that: The user behavior capture module includes: A spatial positioning unit, used to capture the user's real-time location data; The action recognition unit is used to recognize the user's controller operation and gesture action. The behavior data generated by the spatial positioning unit and the action recognition unit are transmitted to the distributed optimal control module.

3. The content dynamic generation and immersive tour interactive system according to claim 1 is characterized in that: The distributed optimal control module includes: A user behavior modeling unit, connected to the user behavior capturing module, for establishing a dynamic behavior distribution based on user location and interaction behavior; A control strategy optimization unit is connected to the user behavior modeling unit and is used to calculate an optimal control strategy for virtual content generation based on the dynamic behavior distribution.

4. The content dynamic generation and immersive tour interactive system according to claim 1 is characterized in that: The user behavior modeling unit describes the user behavior range through a probability distribution function, and transmits the modeling result to the control strategy optimization unit to generate a global control strategy.

5. The content dynamic generation and immersive tour interactive system according to claim 1 is characterized in that: The content generation module comprises: A dynamic content hierarchical loading unit, connected to the distributed optimal control module, is used to preferentially generate high-precision content according to the user's line of sight, and dynamically load the content in the non-line of sight range with low precision; The sparse optimization generation unit is connected to the dynamic content hierarchical loading unit and is used to dynamically generate virtual content through a sparse regularization generation network.

6. The content dynamic generation and immersive tour interactive system according to claim 5 is characterized in that: The sparse optimization generation unit receives the generation priority instruction of the dynamic content classification loading unit, and reduces unnecessary content generation calculations through sparse regularization constraints.

7. The content dynamic generation and immersive tour interactive system according to claim 1 is characterized in that: The physical simulation module comprises: An interactive mechanics modeling unit, connected to the content generation module, for building a multi-body interaction model based on Lagrangian mechanics according to the physical properties of the virtual content and the user's interactive actions; The interactive response calculation unit is connected to the interactive mechanics modeling unit and is used to calculate the dynamic response of the virtual content according to the multi-body interactive model.

8. The content dynamic generation and immersive tour interactive system according to claim 1 is characterized in that: The interactive response calculation unit determines the motion trajectory of the virtual content under the force of the user by calculating the changes in kinetic energy and potential energy of the virtual content, and transmits the calculation result to the multimodal feedback module.

9. The content dynamic generation and immersive tour interactive system according to claim 1, characterized in that: The multimodal feedback module comprises: A visual feedback unit, connected to the physical simulation module, for adjusting the user's visual experience according to the light and shadow changes of the virtual content; A tactile feedback unit, connected to the physical simulation module, for providing a user with an interactive force sense through a vibration module; An audio feedback unit is connected to the physical simulation module and is used to dynamically adjust the sound effects based on the user's interactive position and actions.

10. The content dynamic generation and immersive tour interactive system according to claim 1, characterized in that: It further includes a closed-loop interactive control module, which is connected to the user behavior capture module and the distributed optimal control module and is used to dynamically adjust the global control strategy of content generation based on real-time feedback from users.

Images