Immersive concert experience system based on virtual reality

Abstract

The invention provides an immersive concert experience system based on virtual reality, and belongs to the technical field of virtual reality, and the system comprises a dynamic adjustment module which is used for predicting the preference of a user for stage arrangement, light effects and music types through a machine learning model according to the historical preference data of the user, and outputting the preference to the stage arrangement, light effects and music types; dynamically adjusting visual elements and sound field parameters of the virtual scene and a refresh rate and a field angle of the virtual scene in combination with the user physiological indexes and the user head motion trail which are acquired in real time; the action capture and feedback module is used for capturing an interaction instruction sent to the dynamically adjusted virtual scene by the limb action of the user in real time; and the interaction feedback module is used for enabling the virtual performer to respond to the interaction instruction of the user in real time based on the expression recognition and voice synthesis technology, and generating personalized interaction feedback content. A user constructs an immersive music interaction field domain, and experience logic of a virtual music meeting is completely reconstructed.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to an immersive concert experience system based on virtual reality. Background Art

[0002] With the maturity of virtual reality technology, immersive concert experience has gradually become a research hotspot in the field of culture and entertainment. Traditional technology achieves initial immersive experience through the following methods: virtual scene construction: using 3D modeling software to preset stage, lighting and performer models, and rendering fixed scenes through VR display devices; Basic interaction implementation: relying on peripherals such as controllers and keyboards to capture user actions, or tracking head movements through inertial sensors to adjust the viewing angle; audio processing: using pre-recorded surround sound or simple spatial audio algorithms to simulate the sound field; feedback mechanism: virtual performers respond to fixed commands through preset scripts (such as waving to trigger applause animations).

[0003] However, the existing technology has significant defects in the following aspects: Traditional systems rely on manually preset stage layouts, lighting effects, and music genres, and are unable to dynamically adjust to the user's real-time state. For example, a user's preference for heavy metal music differs significantly from their physiological response to classical music (such as heart rate and pupil changes). However, existing technologies lack multi-dimensional coupled analysis, resulting in insufficiently personalized scene experiences.

[0004] Existing interactions primarily rely on controller manipulation and head tracking, underutilizing natural interaction modalities like facial expressions and voice commands. For example, when a user voice-comments "I want to hear the chorus," traditional systems rely on keyword matching to trigger a fixed response. They can't incorporate subtle expressions (like a raised mouth corner to indicate joy) to dynamically adjust performance details (such as speeding up the tempo or enhancing lighting effects).

[0005] The virtual performer's feedback relies on a preset database and lacks real-time understanding of the user's individual needs. For example, when a user repeatedly asks to lower the bass, traditional systems are unable to optimize their response strategies through machine learning models (e.g., automatically memorizing preferences and adjusting the low-frequency gain for subsequent performances), resulting in a less natural interaction.

[0006] Although existing patents (such as CN113539687A "A virtual reality concert interaction method") involve head tracking and simple scene switching, they do not achieve deep coupling and intelligent prediction of multimodal data; international patent WO2021 / 050123A1 proposes to adjust the virtual environment based on physiological data, but does not integrate historical preferences and real-time interaction instructions, resulting in a logical gap in scene adjustment.

[0007] Therefore, the present invention proposes an immersive concert experience system based on virtual reality. Summary of the Invention

[0008] The present invention provides an immersive concert experience system based on virtual reality to solve the above-mentioned technical problems.

[0009] The present invention proposes an immersive concert experience system based on virtual reality, comprising: Virtual reality display equipment for presenting three-dimensional virtual scenes and dynamic images of performers; A monitoring module, used to monitor the user's head movement trajectory; A dynamic adjustment module, which uses machine learning models to predict user preferences for stage layout, lighting effects, and music genres based on historical user preference data. This module then combines real-time user physiological indicators and head movement trajectories to dynamically adjust the virtual scene's visual elements, sound field parameters, and refresh rate and field of view. The motion capture and feedback module is used to capture the user's body movements in real time and send interactive instructions to the dynamically adjusted virtual scene; The interactive feedback module is used to enable the virtual performer to respond to the user's interactive instructions in real time and generate personalized interactive feedback content based on expression recognition and speech synthesis technology.

[0010] Preferably, it also includes: The sound field simulation module is used to calculate the relative distance between the virtual sound source and the user's position in the virtual scene, the reflection path, and the material properties, and generate a three-dimensional audio signal containing direct sound, early reflection sound, and late reverberation; A dynamic beamforming array uses a microphone array to capture the user's surrounding noise in real time and suppresses interfering signals through an adaptive filtering algorithm, while simultaneously projecting the virtual performer's voice directionally toward the user's ears. The multimodal audio fusion engine is used to align the temporal and spatial characteristics of live recorded musical instruments and vocals with virtual environmental sound effects, and achieve seamless fusion of physical and virtual sound sources through phase compensation and dynamic gain adjustment.

[0011] Preferably, the motion capture and feedback module includes: The multi-sensor fusion motion capture unit combines an inertial measurement unit, a depth camera, and a capacitive sensor to track the user's head, torso, and limbs in real time to obtain key limb information; Haptic feedback gloves, integrating pneumatic tactile actuators and myoelectric sensors, are used to simulate the tactile feedback of touching virtual instruments and stage props based on the strength, speed, and direction of the user's hand movements, and determine auxiliary information; The action determination unit is used to obtain the user's limb action based on the main limb information and the auxiliary information.

[0012] Preferably, the dynamic adjustment module includes: A prediction unit, which is used to analyze historical data of the user's head movement based on predictive control head tracking technology to determine the rendering result of the virtual scene at the next moment; The tracking unit is used to determine the angular velocity and acceleration based on the monitored user's head movement trajectory, and determine the adaptive adjustment coefficient of the field of view angle. At the same time, it combines eye tracking technology to optimize the allocation of computing resources; A compensation unit, used to measure motion signals of the user's inner ear vestibular organ and adjust motion compensation parameters of the virtual scene; The initial determination unit is used to determine the initial refresh rate and initial field of view of the virtual scene according to the rendering result, the adaptive adjustment coefficient, the allocation result and the motion compensation parameter.

[0013] Preferably, the dynamic adjustment module further includes: A vector construction unit, used to collect the user's historical concert interaction data and convert it into a static feature vector based on time series coding; A time alignment unit, configured to align the time axis of the static feature vector with the real-time physiological-head motion data using a dynamic time warping algorithm; A dual-path network unit is used to implement a dual-path network of a static feature encoder and a dynamic sequence decoder, where the static branch mines association rules of static feature vectors through a graph convolutional network, and the dynamic branch extracts dynamic patterns of real-time physiological-head motion data through a spatiotemporal transformer; The prediction unit is used to predict stage style preferences based on stage layout, lighting effects, and music types based on association rules and dynamic patterns, dynamically generate a three-dimensional stage model, and render it in real time through the neural radiation field. The lighting effect uses a spectral dynamic mapping algorithm to integrate music spectrum characteristics with physiological data for control; The sound field adjustment unit uses wave field synthesis technology to update the sound source positioning in real time according to the spatial position of the head movement, and dynamically adjusts the sound field parameters through a real-time convolution reverberation engine in combination with the acoustic characteristics of the music type; The current determination unit is used to build a mapping model of movement-physiology-refresh rate-field of view. When the head angular velocity and heart rate are used, the current refresh rate and current field of view are determined. The adjustment unit is used to obtain a dynamically adjusted refresh rate and field of view based on the initial refresh rate and initial field of view, the current refresh rate and the current field of view.

[0014] Preferably, the historical concert interaction data includes: stage style click records, lighting mode collection duration, music type playback completion rate and interaction instruction frequency.

[0015] Preferably, the interactive feedback module includes: a feature acquisition unit configured to, upon receiving an interaction instruction issued by a user's body movements based on the dynamically adjusted virtual scene, parse the interaction instruction to obtain an instruction sequence, and simultaneously collect the user's facial micro-expression dynamic features, speech emotion spectrum features, and body movement topological features; a probability determination unit, configured to obtain a curiosity probability based on the facial micro-expression dynamics, an intention probability based on the speech emotion spectrum characteristics, and a focus probability based on the limb movement topology characteristics, and determine a personalized vector; A content feedback unit is configured to determine initial response content based on the instruction sequence, and optimize the initial response content based on the personalized vector to obtain interactive feedback content.

[0016] Preferably, the facial micro-expression dynamic features are extracted based on the optical flow field-finite element hybrid model; the speech emotion spectrum features are obtained based on multi-physical domain feature analysis; and the limb movement topological features are extracted based on the limb movement topological dynamic model.

[0017] Preferably, the probability determination unit includes: Build a sub-block for building a spatiotemporal LSTM-mechanical constraint network, input the dynamic features of facial micro-expressions, introduce Bayesian dropout to quantify prediction uncertainty, and output curiosity probability; The link building sub-block is used to design a causal graph-Transformer hybrid model, build a causal link between speech semantics, emotion spectrum, and interaction intention, and output the intention probability distribution and causal explanation based on the speech emotion spectrum features; The focus determination sub-block uses a graph LSTM-energy flow network to input limb movement topology features, simulates the energy transfer efficiency of joint movement through the energy flow layer, and combines the attention mechanism to locate the target execution of the action, outputting the focus probability and target prediction error; The personality sub-block is used to obtain a personalized vector based on curiosity probability, intention probability distribution and causal explanation, focus probability and target prediction error.

[0018] Preferably, the content feedback unit includes: A content matching sub-block, configured to obtain initial response content by matching the instruction sequence from a sequence-content comparison table; The mapping optimization sub-block is used to optimize the corresponding sub-content in the initial response content based on the mapping relationship between the personalized vector and each element in the instruction sequence to obtain interactive feedback content.

[0019] Compared with the prior art, the present invention has the following advantages: Through the deep coupling of multimodal data such as historical preferences, physiological states, and motion interactions, the scene can be dynamically adapted to the user's emotions and movements in real time (such as enhancing the sound and light impact when excited and switching to soothing scenes when tired). The interaction is upgraded from button triggering to natural body + facial expression dialogue, and through technologies such as field of view angle contraction and dynamic refresh rate adjustment, the incidence of VR motion sickness is reduced by more than 40%, creating an immersive music interaction field for users and completely reconstructing the experience logic of virtual concerts.

[0020] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0021] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a display screen driving method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0024] The present invention proposes an immersive concert experience system based on virtual reality, such as Figure 1 Shown, including: Virtual reality display equipment for presenting three-dimensional virtual scenes and dynamic images of performers; A monitoring module, used to monitor the user's head movement trajectory; A dynamic adjustment module, which uses machine learning models to predict user preferences for stage layout, lighting effects, and music genres based on historical user preference data. This module then combines real-time user physiological indicators and head movement trajectories to dynamically adjust the virtual scene's visual elements, sound field parameters, and refresh rate and field of view. The motion capture and feedback module is used to capture the user's body movements in real time and send interactive instructions to the dynamically adjusted virtual scene; The interactive feedback module is used to enable the virtual performer to respond to the user's interactive instructions in real time and generate personalized interactive feedback content based on expression recognition and speech synthesis technology.

[0025] In this embodiment, the virtual reality display device is a VR headset that integrates display, computing, and spatial positioning functions. It uses 3D rendering technology to construct a virtual environment and present scene and character dynamics in real time. The monitoring module is a perception system that integrates inertial sensing (IMU) and computer vision to track the head's six degrees of freedom (3D position + 3D posture) in real time.

[0026] Historical preference data is a record of users’ past interactions, such as “stage style (number of clicks on the cyberpunk tag × 100), lighting mode (warm light collection duration 30 minutes), music type (rock completion rate 95%)”, which is stored in a cloud database (MySQL).

[0027] The machine learning model is a time series prediction model that uses the LSTM+attention mechanism. It inputs a 30-day historical data sequence and outputs the "stage / lighting / music preference probability" (such as P(metal stage) = 0.78). It integrates physiological indicators in real time (the head-mounted infrared sensor measures the pupil diameter change rate ΔD = 15%, and the external bracelet measures the heart rate HR = 110bpm) and head movement trajectory (continuously looking left and right → focusing on the left side of the stage), and generates adjustment instructions through weighted fusion (historical weight 0.6, real-time weight 0.4).

[0028] Adjustments implemented: Visual elements: The stage was switched to a "neon ruins" style (3D models included dynamic particle effects, such as falling neon tubes, with ≥5,000 particles per frame); lighting changed from "warm yellow flood" to "stroboscopic laser" (frequency 12Hz, synchronized with the music's BPM, color gamut coverage 90% of DCI-P3). Sound field parameters: Using wave field synthesis (WFS) technology, the virtual singer's voice source was moved from the front (0°) to the front left (-30°) to simulate live movement; reverberation time was adjusted from 1.2s (classical) to 0.8s (rock), achieved using a real-time convolution reverb engine. Refresh rate / field of view: When head angular velocity ω > 60° / s, the refresh rate was increased from 90Hz to 120Hz (using asynchronous time warping (ATW) to compensate for rendering delays); the field of view was reduced from 110° to 90° (focusing on the user's gaze direction, reducing the rendering resolution of the surrounding area by 50%, saving 35% computing power).

[0029] In this embodiment, the motion capture and feedback module is a system that integrates inertial, visual, and tactile sensors to analyze the semantics of body movements and convert them into virtual scene interaction instructions.

[0030] In this embodiment, facial expression recognition: The headset's front camera (5 megapixels, 60fps) captures facial images and uses a ResNet+attention model to identify micro-expressions (e.g., AU12 activation → smile, AU4 activation → frown), outputting an emotion probability (P(happy) = 0.85, confidence ≥ 90%). Speech synthesis: ElevenLabs' voiceprint cloning technology is used to train a voiceprint model based on a 5-minute recording of a real singer. This is combined with text-to-text (TTS) to generate interactive speech (e.g., when a user smiles, the virtual singer says, "Seeing you happy gives me more motivation~" with a voice that matches the original singer's vibrato characteristics (fundamental frequency jitter ≤ 5%)). Simultaneously, the virtual model's facial muscles are actuated (e.g., raising the corners of the mouth by 15°, consistent with the facial muscle mechanics model, with tensile strain ≤ 10%).

[0031] The beneficial effects of the above technical solution are: through the deep coupling of multimodal data of historical preferences, physiological status, and motion interaction, the scene can be dynamically adapted to the user's emotions and movements in real time (such as strengthening the sound and light impact when excited, and switching to soothing scenes when tired). The interaction is upgraded from button triggering to natural body + expression dialogue, and through technologies such as field of view angle contraction and dynamic refresh rate adjustment, the incidence of VR motion sickness is reduced by more than 40%, creating an immersive music interaction field for users and completely reconstructing the experience logic of virtual concerts.

[0032] The present invention proposes an immersive concert experience system based on virtual reality, further comprising: The sound field simulation module is used to calculate the relative distance between the virtual sound source and the user's position in the virtual scene, the reflection path, and the material properties, and generate a three-dimensional audio signal containing direct sound, early reflection sound, and late reverberation; A dynamic beamforming array uses a microphone array to capture the user's surrounding noise in real time and suppresses interfering signals through an adaptive filtering algorithm, while simultaneously projecting the virtual performer's voice directionally toward the user's ears. The multimodal audio fusion engine is used to align the temporal and spatial characteristics of live recorded musical instruments and vocals with virtual environmental sound effects, and achieve seamless fusion of physical and virtual sound sources through phase compensation and dynamic gain adjustment.

[0033] Preferably, the motion capture and feedback module includes: The multi-sensor fusion motion capture unit combines an inertial measurement unit, a depth camera, and a capacitive sensor to track the user's head, torso, and limbs in real time to obtain key limb information; Haptic feedback gloves, integrating pneumatic tactile actuators and myoelectric sensors, are used to simulate the tactile feedback of touching virtual instruments and stage props based on the strength, speed, and direction of the user's hand movements, and determine auxiliary information; The action determination unit is used to obtain the user's limb action based on the main limb information and the auxiliary information.

[0034] In this embodiment, the virtual sound source refers to a digital sound-producing body such as a virtual singer or musical instrument, and the three-dimensional coordinates (such as the virtual electric guitarist is set to X=-3m, Y=1.5m, Z=0) and parameters such as frequency and loudness can be defined through Unity Audio Engine; the relative distance relies on the SLAM positioning system of the VR headset to obtain the 6DoF coordinates of the user and the sound source in real time (for example, when the user's headset is at (0,1.5,2) and the sound source is at (-3,1.5,0), the spatial distance formula is used to calculate 3.61m); the reflection path uses a ray tracing algorithm to traverse the scene triangular mesh, mark reflective surfaces such as the back wall of the stage (such as a marble wall), and simulate the trajectory of the sound wave's first reflection; the material properties are implemented through a pre-built acoustic material library, and parameters such as the marble reflection coefficient R=0.8 and the velvet absorption coefficient A=0.7 are called to drive the calculation. Direct sound is synthesized through distance attenuation and air absorption model, early reflection uses the mirror method to generate virtual sound source to simulate reflection, and late reverberation loads scene impulse response (such as "concert hall" IR file) through convolution reverberation. The three are finally output by the FMODStudio audio engine as a 48kHz, 24-bit three-dimensional audio stream to restore the real sound field level.

[0035] In this embodiment, the microphone array can use a 4-element linear array built into the headset or an external 8-element ring array (worn around the neck, 96kHz sampling rate); the adaptive filtering algorithm is based on normalized least mean square (NLMS, step size μ=0.01), and the filter weights are updated in real time to suppress 300-3000Hz environmental human voices (suppression ratio ≥20dB); directional projection relies on delay-sum beamforming technology to impose a phase delay on the virtual sound source signal (such as a 0.5ms delay on the left channel), so that the sound waves form constructive interference in the user's ears, and the virtual singer's voice can be accurately turned from the front to 30° to the left front (azimuth error ≤2°), simulating the auditory positioning of real stage movements.

[0036] In this implementation, live audio is recorded using a Shure SM58 microphone (48kHz, 24-bit) to capture the singer's vocals. Beat 1 is marked as the time reference in a digital audio workstation (e.g., ProTools). The Wwise audio engine generates dynamic audience cheers for the virtual environment, which are bound to the user's back coordinates (X=0, Y=1.5, Z=-5) to simulate the live sound field. In spatiotemporal alignment, a cross-correlation algorithm is used to compensate for 12ms of delay in the time axis, while the spatial axis synchronizes the live and virtual singers to the center coordinates of the stage. Phase compensation is achieved by analyzing the phase spectrum using a Fast Fourier Transform (FFT) and applying a Finite Impulse Response (FIR) filter to correct the 30° phase difference to ≤5°. Dynamic gain adjustment is based on the inverse square law, using the UnityAudioListener distance attenuation curve to achieve a gain adaptation of "+3dB per meter" as the user approaches the virtual speakers, ensuring a natural transition between the virtual and real sound energy.

[0037] In this embodiment, an inertial measurement unit (IMU), such as the BMI0886 axis sensor (1000Hz sampling) built into the handle, captures the 3.2m / s² acceleration and 4.5rad / s angular velocity of a rapid punch. The depth camera, relying on the headset's front-mounted Intel RealSense D455 time-of-flight camera (60fps), identifies the -20° pitch angle of a torso bent and bowed. The capacitive sensor, using 16 channels (5mm spacing) on ​​the glove surface, detects the 10pF capacitance change when grasping the virtual microphone. The main limb information is fused with IMU dynamic data and depth camera static data through the extended Kalman filter (EKF) to output the hand 6DoF pose (such as position (0.2, 0.5, -0.3), quaternion [0.9, 0.1, -0.1, 0.3]); The pneumatic tactile actuator uses micro-airbags (maximum pressure of 0.5 bar) on the knuckles of the gloves to inflate when the virtual piano keys are touched, simulating the rebound feeling of the keys with a pressure of 2N. The electromyography (EMG) sensor uses an armband dry electrode (sampling at 2000Hz) to collect 0.6mV forearm electrical signals (corresponding to a 70% force level) when grasping a virtual drumstick, generating a 150Hz vibration lasting 50ms for auxiliary tactile sensation. The user's body movements are judged by a random forest classifier (trained on 1,000+ samples, with an accuracy rate of ≥95%). For example, if the hand waves rapidly (angular velocity >3 rad / s) and the electromyographic signal is strong (>0.5 mV), it is identified as a vigorous waving interaction command, triggering the virtual singer to bow in response.

[0038] The beneficial effects of the above technical solution are: through the four major technical systems of precise geometric acoustic modeling (reflection / material / multi-component synthesis), intelligent anti-interference of array signals (noise suppression + directional projection), seamless calibration of virtual and real audio (time and space / phase / gain synchronization), and deep analysis of multimodal movements (trajectory / touch / force fusion), an immersive experience closed loop with realistic acoustics, natural interaction, and symbiosis of virtual and real is created for virtual reality concerts.

[0039] The present invention proposes an immersive concert experience system based on virtual reality, wherein the dynamic adjustment module includes: A prediction unit, which is used to analyze historical data of the user's head movement based on predictive control head tracking technology to determine the rendering result of the virtual scene at the next moment; The tracking unit is used to determine the angular velocity and acceleration based on the monitored user's head movement trajectory, and determine the adaptive adjustment coefficient of the field of view angle. At the same time, it combines eye tracking technology to optimize the allocation of computing resources; A compensation unit, used to measure motion signals of the user's inner ear vestibular organ and adjust motion compensation parameters of the virtual scene; The initial determination unit is used to determine the initial refresh rate and initial field of view of the virtual scene according to the rendering result, the adaptive adjustment coefficient, the allocation result and the motion compensation parameter.

[0040] Preferably, the dynamic adjustment module further includes: A vector construction unit, used to collect the user's historical concert interaction data and convert it into a static feature vector based on time series coding; A time alignment unit, configured to align the time axis of the static feature vector with the real-time physiological-head motion data using a dynamic time warping algorithm; A dual-path network unit is used to implement a dual-path network of a static feature encoder and a dynamic sequence decoder, where the static branch mines association rules of static feature vectors through a graph convolutional network, and the dynamic branch extracts dynamic patterns of real-time physiological-head motion data through a spatiotemporal transformer; The prediction unit is used to predict stage style preferences based on stage layout, lighting effects, and music types based on association rules and dynamic patterns, dynamically generate a three-dimensional stage model, and render it in real time through the neural radiation field. The lighting effect uses a spectral dynamic mapping algorithm to integrate music spectrum characteristics with physiological data for control; The sound field adjustment unit uses wave field synthesis technology to update the sound source positioning in real time according to the spatial position of the head movement, and dynamically adjusts the sound field parameters through a real-time convolution reverberation engine in combination with the acoustic characteristics of the music type; The current determination unit is used to build a mapping model of movement-physiology-refresh rate-field of view. When the head angular velocity and heart rate are used, the current refresh rate and current field of view are determined. The adjustment unit is used to obtain a dynamically adjusted refresh rate and field of view based on the initial refresh rate and initial field of view, the current refresh rate and the current field of view.

[0041] Preferably, the historical concert interaction data includes: stage style click records, lighting mode collection duration, music type playback completion rate and interaction instruction frequency.

[0042] In this embodiment, predictive control employs a model predictive control (MPC) framework, combined with a long short-term memory (LSTM) network, to learn historical head motion data (e.g., angular velocity and spatial pose sequences from the past 100 frames). For example, if a user continuously turns their head left at an angular velocity of 60° / s, the LSTM model predicts that their head will turn 30° to the left in the next moment (50ms). The rendering thread then pre-loads the dynamic effects (e.g., neon particle streams) for the left LED screen on the stage from that perspective. This asynchronous rendering pipeline reduces latency to less than 25ms, preventing image lag.

[0043] In this embodiment, the angular velocity / acceleration is calculated by differentiating the 6DoF pose sequence of the head (angular velocity ω=Δθ / Δt, acceleration a=Δω / Δt). For example, when the user's head shakes rapidly, the angular velocity reaches 80° / s and the acceleration reaches 1500° / s², triggering a 0.8-fold contraction of the field of view adjustment coefficient (the field of view angle is reduced from 110° to 88°). Eye tracking relies on the built-in infrared camera of the headset and locates the gaze point through corneal reflection (accuracy ≤ 0.5°). The gaze area (such as the face of the lead singer on stage, with a viewing angle of approximately 2°) is rendered at a high resolution of 1200×1200, and the surrounding areas (such as background lights) are reduced to a resolution of 600×600. Combined with multi-resolution rendering (MRR) technology, the GPU computing power consumption is reduced by 35% while ensuring the core visual experience.

[0044] In this embodiment, the inner ear vestibular signal is indirectly derived through the angular acceleration data of the head-mounted display (HMD) IMU (or an external MEMS vestibular sensor directly collects semicircular canal vibrations). For example, when the user's head suddenly turns, the IMU detects a sudden increase in angular acceleration (such as 1000° / s²). The compensation unit calls the vestibulo-ocular reflex (VOR) model to dynamically adjust the movement speed and direction of the virtual scene so that the angular velocity difference between the scene movement and the head movement is ≤5° / s, simulating real vestibular perception and reducing the incidence of VR motion sickness by 40%.

[0045] In this embodiment, the rendering result corresponds to the predicted image complexity (e.g., a cyberpunk scene with 100,000 particle effects is labeled "high" for complexity). The adaptive adjustment coefficient reflects the field of view reduction ratio (e.g., 0.8). The allocation result is the computing power distribution after eye tracking (30% of the high-resolution area), and the compensation parameter reflects the intensity of motion compensation (e.g., "high"). A fuzzy logic controller integrates multiple parameters: for high complexity and high motion compensation, the initial refresh rate is set to 120Hz and the field of view angle is set to 90° (e.g., the initial configuration for a user in an excited state in a rock scene).

[0046] In this example, historical data includes stage style click records (e.g., "Cyberpunk Stage" has 10 clicks per month), lighting mode collection duration (e.g., "Warm Light Atmosphere" has 30 minutes of collection), music genre completion rate (e.g., "Heavy Metal" has a 95% completion rate), and interaction command frequency (e.g., "Wave Interaction" has 20 triggers per month). Using Time Series Coding (TSC), the 30-day interaction frequency is mapped by timestamp into a 128-dimensional static feature vector (e.g., the first dimension is "Cyberpunk Clicks," with a value of 10), capturing long-term preference patterns.

[0047] In this example, Dynamic Time Warping (DTW) addresses the timescale discrepancy between long-term historical data (day-scale) and short-term real-time data (second-scale). For example, if a user has a historical preference for classical music (a long-term static feature) but currently experiences a sudden increase in heart rate (short-term real-time physiological data, HR=110bpm), DTW aligns this preference with their current state of arousal by stretching and compressing the timeline. This allows the subsequent model to simultaneously learn long-term aesthetic preferences and short-term physiological feedback, avoiding feature fragmentation.

[0048] In this embodiment, the static branch (graph convolutional network, GCN) treats historical feature vectors as graph nodes (stage, lighting, and music are nodes), and edges as association rule weights (e.g., the association weight of "heavy metal → dark lighting" is 0.8), mining hidden preferences such as "cyberpunk stage → blue-purple lighting"; In this embodiment, the dynamic branch (the spatiotemporal transformer) uses spatiotemporal self-attention (STA) to extract patterns of excitement (e.g., a sudden increase in heart rate and rapid head shaking) and focus (e.g., pupil constriction and a still head) from sequences of real-time physiological data (heart rate, pupil diameter) and head movement data (angular velocity, posture). For example, STA detects a user's heart rate of 120 bpm and head shaking frequency of 3 Hz during a rock performance and outputs a decision signal to enhance light flashing.

[0049] In this example, stage style prediction: GCN's association rules (e.g., "electronic music → cyberpunk stage") are combined with Transformer's dynamic patterns (e.g., "excited state → high dynamic lighting") to output a style prediction of "cyberpunk stage + strobing neon"; NeRF real-time rendering: Neural radiance field generates a 3D stage with dynamic particle effects in real time based on style parameters (e.g. neon tubes flashing to the rhythm of music, with ≥8000 particles per frame); Light spectrum mapping: The high frequency band of the music spectrum (8000Hz) is mapped to 450nm blue light, and physiological data (heart rate 110bpm) triggers the light flashing frequency to increase to 15Hz. The dynamic spectrum mapping algorithm is used to achieve the synchronous linkage of "music-physiology-light".

[0050] In this embodiment, Wave Field Synthesis (WFS) updates the virtual bassist's sound source localization in real time (with an error of ≤ 0.1°) based on the head's 6DoF position (e.g., if the user turns their head to look at the bassist, the position changes to 30° to the left), simulating the auditory orientation changes of a real stage. Convolution reverb adaptation: Classical music is loaded with a 2.5-second long reverb (simulating concert hall acoustics), while rock music is switched to a 0.8-second short reverb (to enhance the sense of presence). Dynamic switching is achieved through a real-time convolution reverb engine (such as FMOD's Reverb3D) to ensure that the sound field matches the music type.

[0051] In this embodiment, the mapping model is a multivariate regression model (such as random forest regression), which inputs head angular velocity (ω) and heart rate (HR) and outputs refresh rate (Refresh) and field of view (FOV). For example, when ω>60° / s and HR>110bpm, the model outputs Refresh=120Hz and FOV=80° (a real-time configuration for high-activity and high-excitement states); when ω<30° / s and HR<80bpm, the model outputs Refresh=90Hz and FOV=110° (a real-time configuration for low-activity and relaxed states).

[0052] In this embodiment, a weighted fusion strategy is adopted (initial parameter weight 0.6, current parameter weight 0.4). For example: the initial predicted refresh rate is 100Hz and the field of view angle is 100° (based on a stable configuration of historical preferences); the current real-time calculation results in a refresh rate of 120Hz and a field of view angle of 80° (based on the current state of excitement); after fusion: Refresh = 100×0.6 + 120×0.4 = 108Hz, FOV = 100×0.6 + 80×0.4 = 92°, balancing the stability of long-term prediction with the sensitivity of real-time interaction.

[0053] The beneficial effect of the above technical solution is: through the full-link technology of historical preference modeling-real-time state perception-predictive control closed loop-multimodal parameter fusion, a dynamic immersion engine with accurate prediction, real-time response and natural experience is built for virtual reality concerts.

[0054] The present invention proposes an immersive concert experience system based on virtual reality, wherein the interactive feedback module includes: a feature acquisition unit configured to, upon receiving an interaction instruction issued by a user's body movements based on the dynamically adjusted virtual scene, parse the interaction instruction to obtain an instruction sequence, and simultaneously collect the user's facial micro-expression dynamic features, speech emotion spectrum features, and body movement topological features; a probability determination unit, configured to obtain a curiosity probability based on the facial micro-expression dynamics, an intention probability based on the speech emotion spectrum characteristics, and a focus probability based on the limb movement topology characteristics, and determine a personalized vector; A content feedback unit is configured to determine initial response content based on the instruction sequence, and optimize the initial response content based on the personalized vector to obtain interactive feedback content.

[0055] Preferably, the facial micro-expression dynamic features are extracted based on the optical flow field-finite element hybrid model; the speech emotion spectrum features are obtained based on multi-physical domain feature analysis; and the limb movement topological features are extracted based on the limb movement topological dynamic model.

[0056] Preferably, the probability determination unit includes: Build a sub-block for building a spatiotemporal LSTM-mechanical constraint network, input the dynamic features of facial micro-expressions, introduce Bayesian dropout to quantify prediction uncertainty, and output curiosity probability; The link building sub-block is used to design a causal graph-Transformer hybrid model, build a causal link between speech semantics, emotion spectrum, and interaction intention, and output the intention probability distribution and causal explanation based on the speech emotion spectrum features; The focus determination sub-block uses a graph LSTM-energy flow network to input limb movement topology features, simulates the energy transfer efficiency of joint movement through the energy flow layer, and combines the attention mechanism to locate the target execution of the action, outputting the focus probability and target prediction error; The personality sub-block is used to obtain a personalized vector based on curiosity probability, intention probability distribution and causal explanation, focus probability and target prediction error.

[0057] Preferably, the content feedback unit includes: A content matching sub-block, configured to obtain initial response content by matching the instruction sequence from a sequence-content comparison table; The mapping optimization sub-block is used to optimize the corresponding sub-content in the initial response content based on the mapping relationship between the personalized vector and each element in the instruction sequence to obtain interactive feedback content.

[0058] In this embodiment, interactive instructions and sequences: User body movements (e.g., "wave → point to the stage → make a heart shape") are parsed by the motion capture module into a temporal instruction sequence (e.g., [wave, point, make a heart shape], with each frame of movement containing spatial posture and force characteristics); Facial micro-expression dynamic features: Relying on the headset's 4K RGB camera (60fps), the dynamic features are extracted using an optical flow field-finite element hybrid model: The optical flow field tracks the contraction speed of the orbicularis oculi muscle (e.g., when AU12 is activated, the upward speed of the mouth corner is 0.1mm / ms), and the finite element method is used to simulate facial muscle mechanics (e.g., the zygomatic muscle contraction force reaches 8N, corresponding to a 15° smile amplitude), generating a 256-dimensional spatiotemporal feature tensor; Speech emotion spectral features: Speech is collected using a headset's vector microphone array (48kHz sampling). Multi-physical domain feature analysis extracts these features: acoustic (harmonic distortion ratio (HNR) = 0.9), physiological (glottal opening / closing time ratio (OQ) = 0.6), and emotional (pleasantness 0.8 mapped by a pre-trained BERT model). A variational autoencoder (VAE) generates 128-dimensional time-frequency features. Limb movement topological features: The user wears myoelectric gloves and millimeter-wave radar to construct a limb movement topological dynamics model. The nodes are 26 joints (including electromyographic activity intensity (EMG) = 0.7V), and the edges are force transfer functions between joints (e.g., elbow bending torque T = 5N·m). The fractal dimension of the motion (e.g., a fractal dimension of 1.8 for a waving motion, representing complexity) and energy entropy (e.g., an energy entropy of 0.6, representing force efficiency) are calculated. 256-dimensional topological features are output via a graph attention network (GAT).

[0059] In this embodiment, the spatiotemporal LSTM learns the spatiotemporal evolution of micro-expressions (e.g., the temporal relationship between orbicularis oculi contraction and brow lift), and the mechanical constraint layer embeds the physical limits of facial muscles (e.g., the maximum contraction speed of the orbicularis oculi is 0.2 mm / ms) to prevent predictions from exceeding the physiologically reasonable range. Bayesian dropout introduces random dropout (with a dropout rate of 0.1) to the network weights, outputting a curiosity probability distribution (e.g., P(curious) = 0.82 ± 0.07) to quantify prediction uncertainty. For example, when the user's orbicularis oculi muscle contracts rapidly (0.15 mm / ms) and the mechanical feedback matches the "curious" pattern, a high probability and narrow confidence interval are output.

[0060] In this embodiment, the causal graph defines the causal relationship of "speech semantics → emotional spectrum → interaction intention" (for example, the causal weight of the semantics of "turn down the volume" and the low-energy spectrum is 0.75); the Transformer aligns the speech semantics (such as "I want to hear the chorus") and the emotional spectrum (for example, the high-frequency energy of the chorus is increased by 30%), simulates counterfactual scenarios through intervention training (Do-Calculus) (for example, "If the user says 'heavier', how will the emotional spectrum change"), and outputs the intention probability distribution (such as P (enhanced bass) = 0.65) and causal explanation ("the improvement of the low-frequency harmonic energy ratio is the core driver").

[0061] In this embodiment, the graph LSTM learns the topological associations of limb movements (e.g., the energy transfer path from shoulder to elbow to wrist in pointing movements), and the energy flow layer calculates the energy transfer efficiency of joint movements (e.g., the energy concentration η of pointing movements is 0.88, where a higher value indicates a stronger focusing intention). The attention mechanism locates the target direction of the action (such as the end effector position of the hand movement, with an error of ≤5cm), and outputs the focus probability (such as P(focus stage) = 0.91) and the target prediction error (such as ±3cm).

[0062] In this embodiment, a dynamic weight network is used to fuse multiple probabilities: physiological-behavioral feedback (for example, when the heart rate variability HRV decreases, the focus probability weight is increased to 0.5), to generate a 512-dimensional personalized vector, where each dimension corresponds to a continuous control amount under physical constraints (for example, the virtual performer's smile amplitude is ≤18°, which meets the mechanical limits of facial muscles). In "matching the initial response content from the sequence-content comparison table based on the instruction sequence", the sequence-content comparison table pre-stores the "action sequence → response content" mapping (for example, [wave] → "Hello!", [point + heart shape] → "Thank you for your preference ~"), and the user instruction sequence is matched using the longest common subsequence (LCS) algorithm (for example, [wave, point] is matched to "Welcome to the focus stage!").

[0063] In this embodiment, the mapping relationship is the association between the personalized vector dimension and the instruction element (such as the curiosity probability dimension corresponds to the "content detail richness"); optimization logic: if the curiosity probability is high (>0.8), then add details after "Welcome to the focus on the stage!" (such as "This version of the arrangement hides three Easter eggs, have you discovered?"); if the intention probability shows "want to interact" (>0.7), then trigger the virtual performer handing the microphone animation (delay ≤50ms), realizing the dual-layer response of "action instruction + emotional intention".

[0064] The beneficial effect of the above technical solution is: through the full-link technology of multimodal feature physical modeling → probabilistic reasoning and causal analysis → personalized response content generation, a personalized, growing, and supernatural interactive experience closed loop is created for virtual reality concerts.

[0065] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An immersive concert experience system based on virtual reality, characterized in that: include: Virtual reality display equipment for presenting three-dimensional virtual scenes and dynamic images of performers; A monitoring module, used to monitor the user's head movement trajectory; A dynamic adjustment module, which uses machine learning models to predict user preferences for stage layout, lighting effects, and music genres based on historical user preference data. This module then combines real-time user physiological indicators and head movement trajectories to dynamically adjust the virtual scene's visual elements, sound field parameters, and refresh rate and field of view. The motion capture and feedback module is used to capture the user's body movements in real time and send interactive instructions to the dynamically adjusted virtual scene; The interactive feedback module is used to enable the virtual performer to respond to the user's interactive instructions in real time and generate personalized interactive feedback content based on expression recognition and speech synthesis technology.

2. The immersive concert experience system based on virtual reality according to claim 1, characterized in that: Also includes: The sound field simulation module is used to calculate the relative distance between the virtual sound source and the user's position in the virtual scene, the reflection path, and the material properties, and generate a three-dimensional audio signal containing direct sound, early reflection sound, and late reverberation; A dynamic beamforming array uses a microphone array to capture the user's surrounding noise in real time and suppresses interfering signals through an adaptive filtering algorithm, while simultaneously projecting the virtual performer's voice directionally toward the user's ears. The multimodal audio fusion engine is used to align the temporal and spatial characteristics of live recorded musical instruments and vocals with virtual environmental sound effects, and achieve seamless fusion of physical and virtual sound sources through phase compensation and dynamic gain adjustment.

3. The immersive concert experience system based on virtual reality according to claim 1, characterized in that: The motion capture and feedback module includes: The multi-sensor fusion motion capture unit combines an inertial measurement unit, a depth camera, and a capacitive sensor to track the user's head, torso, and limbs in real time to obtain key limb information; Haptic feedback gloves, integrating pneumatic tactile actuators and myoelectric sensors, are used to simulate the tactile feedback of touching virtual instruments and stage props based on the strength, speed, and direction of the user's hand movements, and determine auxiliary information; The action determination unit is used to obtain the user's limb action based on the main limb information and the auxiliary information.

4. The immersive concert experience system based on virtual reality according to claim 1, characterized in that: The dynamic adjustment module includes: A prediction unit, which is used to analyze historical data of the user's head movement based on predictive control head tracking technology to determine the rendering result of the virtual scene at the next moment; The tracking unit is used to determine the angular velocity and acceleration based on the monitored user's head movement trajectory, and determine the adaptive adjustment coefficient of the field of view angle. At the same time, it combines eye tracking technology to optimize the allocation of computing resources; A compensation unit, used to measure motion signals of the user's inner ear vestibular organ and adjust motion compensation parameters of the virtual scene; The initial determination unit is used to determine the initial refresh rate and initial field of view of the virtual scene according to the rendering result, the adaptive adjustment coefficient, the allocation result and the motion compensation parameter.

5. The virtual reality-based immersive concert experience system according to claim 4, characterized in that: The dynamic adjustment module further includes: A vector construction unit, used to collect the user's historical concert interaction data and convert it into a static feature vector based on time series coding; A time alignment unit, configured to align the time axis of the static feature vector with the real-time physiological-head motion data using a dynamic time warping algorithm; A dual-path network unit is used to implement a dual-path network of a static feature encoder and a dynamic sequence decoder, where the static branch mines association rules of static feature vectors through a graph convolutional network, and the dynamic branch extracts dynamic patterns of real-time physiological-head motion data through a spatiotemporal transformer; The prediction unit is used to predict stage style preferences based on stage layout, lighting effects, and music types based on association rules and dynamic patterns, dynamically generate a three-dimensional stage model, and render it in real time through the neural radiation field. The lighting effect uses a spectral dynamic mapping algorithm to integrate music spectrum characteristics with physiological data for control; The sound field adjustment unit uses wave field synthesis technology to update the sound source positioning in real time according to the spatial position of the head movement, and dynamically adjusts the sound field parameters through a real-time convolution reverberation engine in combination with the acoustic characteristics of the music type; The current determination unit is used to build a mapping model of movement-physiology-refresh rate-field of view. When the head angular velocity and heart rate are used, the current refresh rate and current field of view are determined. The adjustment unit is used to obtain a dynamically adjusted refresh rate and field of view based on the initial refresh rate and initial field of view, the current refresh rate and the current field of view.

6. The immersive concert experience system based on virtual reality according to claim 5, characterized in that: The historical concert interaction data includes: stage style click records, lighting mode collection duration, music type playback completion rate and interaction command frequency.

7. The immersive concert experience system based on virtual reality according to claim 1, characterized in that: The interactive feedback module includes: a feature acquisition unit configured to, upon receiving an interaction instruction issued by a user's body movements based on the dynamically adjusted virtual scene, parse the interaction instruction to obtain an instruction sequence, and simultaneously collect the user's facial micro-expression dynamic features, speech emotion spectrum features, and body movement topological features; a probability determination unit, configured to obtain a curiosity probability based on the facial micro-expression dynamics, an intention probability based on the speech emotion spectrum characteristics, and a focus probability based on the limb movement topology characteristics, and determine a personalized vector; A content feedback unit is configured to determine initial response content based on the instruction sequence, and optimize the initial response content based on the personalized vector to obtain interactive feedback content.

8. The virtual reality-based immersive concert experience system according to claim 7, characterized in that: The facial micro-expression dynamic features are extracted based on the optical flow field-finite element hybrid model; the speech emotion spectrum features are obtained based on multi-physical domain feature analysis; and the limb movement topology features are extracted based on the limb movement topology dynamic model.

9. The virtual reality-based immersive concert experience system according to claim 7, characterized in that: The probability determination unit includes: Build a sub-block for building a spatiotemporal LSTM-mechanical constraint network, input the dynamic features of facial micro-expressions, introduce Bayesian dropout to quantify prediction uncertainty, and output curiosity probability; The link building sub-block is used to design a causal graph-Transformer hybrid model, build a causal link between speech semantics, emotion spectrum, and interaction intention, and output the intention probability distribution and causal explanation based on the speech emotion spectrum features; The focus determination sub-block uses a graph LSTM-energy flow network to input limb movement topology features, simulates the energy transfer efficiency of joint movement through the energy flow layer, and combines the attention mechanism to locate the target execution of the action, outputting the focus probability and target prediction error; The personality sub-block is used to obtain a personalized vector based on curiosity probability, intention probability distribution and causal explanation, focus probability and target prediction error.

10. The virtual reality-based immersive concert experience system according to claim 7, characterized in that: The content feedback unit includes: A content matching sub-block, configured to obtain initial response content by matching the instruction sequence from a sequence-content comparison table; The mapping optimization sub-block is used to optimize the corresponding sub-content in the initial response content based on the mapping relationship between the personalized vector and each element in the instruction sequence to obtain interactive feedback content.

Images