Stage acousto-optic interaction method and system based on virtual reality

Abstract

The invention relates to the technical field of virtual reality, and discloses a stage acousto-optic interaction method and system based on virtual reality, and the method comprises the steps: firstly carrying out the multi-modal context vectorization of a real-time audio stream, performance time axis data and historical user data, so as to obtain a context vector set; meanwhile, user behavior feature extraction is carried out on the original sensor data flow of the user to obtain a user behavior vector; context attention fusion is then performed on the set of context vectors to generate a modulation parameter pair reflecting the current comprehensive context. Then, based on the modulation parameter pair, performing feature-level modulation on the user behavior vector, and decoding to generate an effect parameter vector; and finally, analyzing the effect parameter vector and converting the effect parameter vector into a rendering instruction which can be executed by a rendering engine. Therefore, the acousto-optic feedback on the user behavior can be dynamically adjusted according to the performance context, the accurate matching of the interaction effect and the stage art atmosphere is realized, and the immersion of the user and the art expressive force of the interaction experience are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual reality, and more specifically, to a stage sound and light interaction method and system based on virtual reality. BACKGROUND

[0002] With the continuous progress of virtual reality (VR) technology, its application has expanded from traditional game entertainment to online virtual concerts, online music festivals and other new digital performance scenarios. In these scenarios, improving the immersion and interactivity of users is one of the core technical goals. Unlike the traditional one-way content consumption mode, the virtual reality environment provides the possibility for the audience to influence the performance environment in real time through their own behavior. Therefore, building an interactive system that can map audience behavior in real time to stage sound and light effects has important value for enhancing user engagement and enriching the artistic forms of performance, and has become an important research direction in this field.

[0003] To achieve the above goal, the prior art usually adopts a static mapping scheme based on preset rules. This type of scheme defines a fixed correspondence between user-specific behavior and specific sound and light effects in advance, for example, a certain hand waving action of the user triggers a preset particle special effect. However, this static mapping mechanism has significant technical defects: its interaction logic is irrelevant to the real-time context of the performance. In actual performance activities, the atmosphere and rhythm of the performance are dynamically changing, for example, a song may contain a slow and emotional verse part and a rhythm-intensive chorus part. Under the static mapping mechanism, the user performs the same action in the verse and chorus parts, and the system triggers exactly the same sound and light effects, resulting in a situation where the effects produced do not match the current music mood and stage atmosphere, thereby destroying the artistic integrity of the performance and reducing the immersive experience of the user. The reason for this is that the prior art lacks an effective mechanism for perceiving and utilizing performance context information (such as real-time audio features, performance chapter flow, etc.), and cannot dynamically adjust the mapping logic of user behavior to sound and light effects according to the context, which constitutes a technical problem that needs to be solved in the current technical field. SUMMARY

[0004] In view of the technical problems in the prior art that the interaction mapping relationship is static and lacks dynamic perception of the performance context, the embodiments of the present application propose a stage sound and light interaction method and system based on virtual reality, which can dynamically adjust the interaction mapping logic according to the performance situation to solve the problem of mismatch between the sound and light effects produced by user interaction and the artistic atmosphere of the stage.

[0005] According to an aspect of the present application, a virtual reality-based stage sound-light interaction method is provided, comprising: performing multi-modal context vectorization on acquired real-time audio stream, performance timeline data and historical user data to obtain a set of context vectors; performing user behavior feature extraction on a first user's raw sensor data stream to obtain a first user behavior vector; performing context attention fusion and modulation parameter generation on the set of context vectors to obtain a pair of modulation parameters; based on the pair of modulation parameters, performing feature-level modulation and effect parameter decoding on the first user behavior vector to obtain an effect parameter vector; and performing vector analysis and rendering instruction conversion on the effect parameter vector to obtain rendering instructions.

[0006] In a possible implementation, the multi-modal context vectorization on the acquired real-time audio stream, performance timeline data and historical user data to obtain the set of context vectors comprises: performing music feature extraction on the real-time audio stream to obtain a music context vector; performing virtual-real state encoding on the performance timeline data to obtain a narrative context vector; extracting crowd atmosphere features from the historical user data to obtain a crowd atmosphere context vector; and packing the music context vector, the narrative context vector and the crowd atmosphere context vector to obtain the set of context vectors.

[0007] In a possible implementation, the virtual-real state encoding on the performance timeline data to obtain the narrative context vector comprises: determining a performance chapter to which a current time stamp belongs according to a position of the current time stamp in the performance timeline data; and performing one-hot encoding on the performance chapter to which the current time stamp belongs to obtain the narrative context vector.

[0008] In a possible implementation, the user behavior feature extraction on the first user's raw sensor data stream to obtain the first user behavior vector comprises: performing real-time solution of kinematic basic features on the first user's raw sensor data stream to obtain a kinematic feature time series, each frame of data in the raw sensor data stream being 6DoF posture data; performing spatio-temporal high-order feature aggregation on the kinematic feature time series to obtain an aggregated scalar feature set; and performing vector construction and normalization on the aggregated scalar feature set to obtain the first user behavior vector.

[0009] In a possible implementation, the spatio-temporal high-order feature aggregation on the kinematic feature time series to obtain the aggregated scalar feature set comprises: performing trajectory geometry feature calculation on the kinematic feature time series to obtain a hand trajectory length; performing spatial occupancy feature calculation on the kinematic feature time series to obtain a hands active space volume; and performing motion statistical feature calculation on the kinematic feature time series to obtain an average speed scalar and a motion jitter degree, wherein the hand trajectory length, the hands active space volume, the average speed scalar and the motion jitter degree constitute the aggregated scalar feature set.

[0010] In a possible implementation, the context attention fusion on the context vector set and the modulation parameter generation are performed to obtain a modulation parameter pair, including: performing context nonlinear fusion on the context vector set based on a self-attention mechanism to obtain a fused context vector; and inputting the fused context vector into a parameter generation network to obtain the modulation parameter pair.

[0011] In a possible implementation, the context nonlinear fusion on the context vector set based on the self-attention mechanism is performed to obtain a fused context vector, including: projecting the context vector set into a query space, a key space and a value space to obtain a query matrix, a key matrix and a value matrix; performing context interaction on the query matrix, the key matrix and the value matrix to obtain an interacted context matrix; and performing information aggregation on the interacted context matrix to obtain the fused context vector.

[0012] In a possible implementation, the feature-level modulation on the first user behavior vector and the effect parameter decoding are performed based on the modulation parameter pair to obtain an effect parameter vector, including: inputting the first user behavior vector into a backbone neural network to obtain an intermediate user behavior feature vector; performing feature-level affine transformation on the intermediate user behavior feature vector based on the modulation parameter pair to obtain a modulated user behavior feature vector; and inputting the modulated user behavior feature vector into a decoder to obtain the effect parameter vector.

[0013] In a possible implementation, the vector analysis and the rendering instruction conversion are performed on the effect parameter vector to obtain a rendering instruction, including: performing vector decoding and semantic structuring on the effect parameter vector to obtain a semantic effect descriptor; and performing rendering resource mapping and instruction parameterization on the semantic effect descriptor to obtain the rendering instruction.

[0014] According to another aspect of the present application, a virtual reality-based stage sound-light interaction system is provided, including: a multi-modal context encoding module configured to perform multi-modal context vectorization on acquired real-time audio stream, performance timeline data and historical user data to obtain a context vector set; a user behavior feature extraction module configured to perform user behavior feature extraction on original sensor data stream of a first user to obtain a first user behavior vector; a context attention modulation module configured to perform context attention fusion on the context vector set and modulation parameter generation to obtain a modulation parameter pair; a feature modulation and effect decoding module configured to perform feature-level modulation on the first user behavior vector and effect parameter decoding based on the modulation parameter pair to obtain an effect parameter vector; and a rendering instruction generation module configured to perform vector analysis and rendering instruction conversion on the effect parameter vector to obtain a rendering instruction.

[0015] Compared with the prior art, the virtual reality-based stage sound and light interaction method and system provided by the application firstly overcomes the defect of fixed interaction mapping logic in the prior art, realizes dynamic self-adaptation of interaction response, and enables sound and light effect feedback for the same user behavior to be dynamically adjusted according to real-time changes of a performance situation, thereby guaranteeing high consistency of the interaction effect and the performance atmosphere. Secondly, the artistic expressiveness of virtual performance and the immersive experience of a user are significantly enhanced, because the sound and light effect can be accurately matched with context information such as music emotion and performance rhythm, and the interaction behavior of the user can be harmoniously integrated into the overall artistic expression rather than forming interference. Finally, the interaction system can effectively process and utilize complex and multi-dimensional context information, thereby improving the intelligent level and environmental adaptability of interaction, and providing a user with a more in-depth and meaningful interactive experience. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures. The drawings provided in connection with the present application are intended to further explain, by way of non-limiting example, the principles of the present application, and constitute a part of this specification. The drawings provided in connection with the present application do not constitute an entire specification, and are used to explain the present application together with the present application, and do not constitute a limitation on the present application. In the drawings, the same reference numerals represent the same components or steps throughout the figures.

[0017] Figure 1 FIG. 1 illustrates a schematic flowchart of a virtual reality-based stage sound and light interaction method according to an embodiment of the present application.

[0018] Figure 2 FIG. 1 illustrates a schematic flowchart of a virtual reality-based stage sound and light interaction method according to an embodiment of the present application.

[0019] Figure 3 FIG. 1 illustrates a schematic flowchart of a virtual reality-based stage sound and light interaction method according to an embodiment of the present application.

[0020] Figure 4 FIG. 1 illustrates a schematic flowchart of a virtual reality-based stage sound and light interaction method according to an embodiment of the present application.

[0021] Figure 5 FIG. 1 illustrates a schematic flowchart of a virtual reality-based stage sound and light interaction method according to an embodiment of the present application.

[0022] Figure 6 FIG. 1 illustrates a schematic flowchart of a virtual reality-based stage sound and light interaction method according to an embodiment of the present application.

[0023] Figure 7 FIG. 1 illustrates a schematic flowchart of a virtual reality-based stage sound and light interaction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part but not all of the embodiments of the present application, and the present application can be implemented in many different forms. Therefore, the present application should not be limited to these embodiments described herein, but should be understood to cover all the modifications and alternatives falling within the scope of the present application. In order to clearly describe technical solutions of the present application, the following description will briefly introduce the drawings used in the present application. The following mainly introduces the drawings used in the present application.

[0025] Figure 1 Fig. 1 illustrates a schematic flow chart of a virtual reality-based stage sound-light interaction method and system according to an embodiment of the present application. Figure 2 Fig. 2 illustrates a data flow schematic diagram of a virtual reality-based stage sound-light interaction method according to an embodiment of the present application. As shown in Fig. 2, the present application provides a virtual reality-based stage sound-light interaction method, including: S1, performing multi-modal context vectorization on acquired real-time audio stream, performance timeline data and historical user data to obtain a context vector set; S2, performing user behavior feature extraction on a first user's original sensor data stream to obtain a first user behavior vector; S3, performing context attention fusion and modulation parameter generation on the context vector set to obtain a modulation parameter pair; S4, based on the modulation parameter pair, performing feature-level modulation and effect parameter decoding on the first user behavior vector to obtain an effect parameter vector; and S5, performing vector analysis and rendering instruction conversion on the effect parameter vector to obtain a rendering instruction. Figure 1 And Figure 2 As shown in Fig. 2, the present application provides a virtual reality-based stage sound-light interaction method, including: S1, performing multi-modal context vectorization on acquired real-time audio stream, performance timeline data and historical user data to obtain a context vector set; S2, performing user behavior feature extraction on a first user's original sensor data stream to obtain a first user behavior vector; S3, performing context attention fusion and modulation parameter generation on the context vector set to obtain a modulation parameter pair; S4, based on the modulation parameter pair, performing feature-level modulation and effect parameter decoding on the first user behavior vector to obtain an effect parameter vector; and S5, performing vector analysis and rendering instruction conversion on the effect parameter vector to obtain a rendering instruction.

[0026] Exemplarily, in step S1, multi-modal context vectorization is performed on acquired real-time audio stream, performance timeline data and historical user data to obtain a context vector set. It can be understood that in virtual stage interaction, a proper sound-light effect not only depends on the user's behavior itself, but also depends on the specific situation when the behavior occurs. Therefore, by uniformly processing context information from different sources into a numerical vector, an accurate input basis can be provided for subsequent interaction mapping.

[0027] In one embodiment, as shown in Fig. 1, the multi-modal context vectorization performed on the acquired real-time audio stream, performance timeline data and historical user data to obtain the context vector set includes: S11, performing music feature extraction on the real-time audio stream to obtain a music context vector; S12, performing virtual-real state encoding on the performance timeline data to obtain a narrative context vector; S13, extracting crowd atmosphere features from the historical user data to obtain a crowd atmosphere context vector; and S14, packaging the music context vector, the narrative context vector and the crowd atmosphere context vector to obtain the context vector set. Figure 3

[0028] ​Specifically, first, the acquired real-time audio stream is subjected to music feature extraction to obtain a music context vector. This process extracts key acoustic features that can represent music characteristics from the real-time audio stream through digital signal processing. For example, Mel-frequency cepstral coefficients (MFCCs) are calculated to represent the timbre of the music, chroma features are extracted to represent the harmony and tonality of the music, the beat is analyzed to obtain the beats per minute (BPM) to quantify the speed of the music, and the root mean square energy (RMS) of the signal is calculated to represent the loudness or energy intensity of the music. These extracted numerical features are combined into a fixed-dimensional numerical vector, i.e., the music context vector, to quantify the style, rhythm, and emotion of the current music.

[0029] Second, the performance timeline data is subjected to virtual-real state encoding to obtain a narrative context vector. This process includes first determining the current performance chapter to which the current timestamp belongs in the performance timeline data, such as the verse, chorus, or bridge. Then, the current performance chapter is one-hot encoded to obtain the narrative context vector. This encoding method generates a numerical vector whose dimension is equal to the total number of all possible performance chapters, and only the dimension component corresponding to the current chapter is 1, and all other dimension components are 0. In this way, the narrative stage of the current performance is represented in a non-ambiguous numerical form, thereby obtaining the narrative context vector.

[0030] Third, crowd atmosphere features are extracted from historical user data to obtain a crowd atmosphere context vector. This process analyzes all historical user data within a recent time window to calculate statistical indicators that reflect the overall activity of the group. For example, the average kinetic energy of all users is calculated to quantify the overall movement intensity of the audience; at the same time, the entropy of user behavior patterns can also be calculated to measure the diversity or consistency of user behavior. These statistical indicators are combined into a crowd atmosphere context vector to quantify the overall mood and engagement of the audience.

[0031] Finally, the music context vector, narrative context vector, and crowd atmosphere context vector obtained from the foregoing processing are packaged together to form a structured data set, i.e., the context vector set finally output by this step.

[0032] In a specific embodiment, in the virtual concert scenario, when the performance proceeds to the chorus stage (timestamp t=90 seconds), the music rhythm accelerates, the drum sound is strong, and most of the audience starts to jump vigorously with the rhythm. At this time, the step is executed again, and the obtained music context vector is updated as follows: the BPM is 160, the Energy is 0.9, the chroma vector is updated as [1.0, 0.2, 0.2, 0.3, 0.9, 0.4, 0.2, 0.9, 0.2, 0.3, 0.2, 0.2], which indicates that the harmony structure remains the same but the intensity and richness increase, and the first mel-frequency cepstral coefficient is changed to 3.8, which indicates that the timbre is brighter and more intense. The narrative context vector is updated as a four-dimensional vector, in which the second component is 1 and the remaining components are 0, indicating that the current chapter is the chorus. The crowd atmosphere context vector is updated as follows: the average kinetic energy is 0.8 units, and the behavior entropy value increases to 1.6, indicating that the overall activity of the crowd is high and the behavior mode is more diversified. The three updated vectors are packaged to form the context vector set of the chorus stage.

[0033] When the performance proceeds to the chorus stage (timestamp t=90 seconds), the music rhythm accelerates, the drum sound is strong, and most of the audience starts to jump vigorously with the rhythm. At this time, the step is executed again, and the obtained music context vector is updated as follows: the BPM is 160, the Energy is 0.9, the chroma vector is updated as [1.0, 0.2, 0.2, 0.3, 0.9, 0.4, 0.2, 0.9, 0.2, 0.3, 0.2, 0.2], which indicates that the harmony structure remains the same but the intensity and richness increase, and the first mel-frequency cepstral coefficient is changed to 3.8, which indicates that the timbre is brighter and more intense. The narrative context vector is updated as a four-dimensional vector, in which the second component is 1 and the remaining components are 0, indicating that the current chapter is the chorus. The crowd atmosphere context vector is updated as follows: the average kinetic energy is 0.8 units, and the behavior entropy value increases to 1.6, indicating that the overall activity of the crowd is high and the behavior mode is more diversified. The three updated vectors are packaged to form the context vector set of the chorus stage.

[0034] Exemplarily, in step S2, user behavior feature extraction is performed on the raw sensor data stream of the first user to obtain a first user behavior vector. It should be understood that the raw sensor data is a high-frequency, high-dimensional time series signal, which is not suitable for direct input into the upper-layer algorithm model. Therefore, through user behavior feature extraction, the raw sensor data stream from the first user's VR device end, which is continuous and contains time series information, is converted into a feature vector that can quantitatively describe the user's behavior pattern in a specific time window.

[0035] In one embodiment, as shown in Figure 4 user behavior feature extraction is performed on the raw sensor data stream of the first user to obtain a first user behavior vector, including: S21, performing real-time kinematic basic feature calculation on the raw sensor data stream of the first user to obtain a kinematic feature time series; S22, performing spatio-temporal high-order feature aggregation on the kinematic feature time series to obtain an aggregated scalar feature set; and S23, performing vector construction and normalization on the aggregated scalar feature set to obtain the first user behavior vector.

[0036] Specifically, first, real-time kinematic basic feature calculation is performed. This process takes the raw sensor data stream of the first user as input, where each frame of data in the raw sensor data stream is 6DoF pose data, i.e., data containing three-dimensional spatial position and three-dimensional rotation pose. The data in a preset time window is processed, and by differentiating the position and pose data between consecutive frames, instantaneous linear velocity, linear acceleration, and angular velocity of each tracking point (such as the head and dual-hand controllers) are obtained, thereby forming a kinematic feature time series.

[0037] Then, spatio-temporal high-order feature aggregation is performed on the kinematic feature time series. This process aims to aggregate the time series feature data into a set of scalar values that can describe the overall motion characteristics in the time window. Specifically, this process includes: trajectory geometry feature calculation on the kinematic feature time series, such as calculating the Euclidean distance between consecutive position points to obtain the hand trajectory length; spatial occupancy feature calculation on the kinematic feature time series, such as calculating the convex hull of the hand motion trajectory point set to obtain the hand activity space volume; and motion statistical feature calculation on the kinematic feature time series, such as calculating the mean of the velocity vector magnitude to obtain the average velocity scalar and calculating the standard deviation of the acceleration vector magnitude to obtain the motion jitter degree. Finally, the hand trajectory length, hand activity space volume, average velocity scalar, and motion jitter degree are combined to form an aggregated scalar feature set.

[0038] Finally, the aggregated scalar feature set is vectorized and normalized to obtain the first user behavior vector. This process concatenates all scalar values in the aggregated scalar feature set in a predefined order to form an original high-dimensional vector, and then scales each component in the vector to a uniform numerical interval through mathematical methods such as min-max normalization, so as to eliminate the dimensional influence between different features, thereby obtaining the first user behavior vector as the final output of this step.

[0039] In a specific embodiment, in a virtual concert scenario, the first user performs a "forceful waving from left to right" action lasting about 0.5 seconds. First, the 6DoF pose data stream of the right hand controller in this time window is captured, and kinematic basic feature real-time calculation is performed to obtain a time series containing information such as speed and acceleration. Then, the time series is subjected to spatio-temporal high-order feature aggregation to calculate an aggregated scalar feature set, the specific values of which can be: hand trajectory length 1.2 meters, double-hand activity space volume 0.5 cubic meters, average speed scalar 2.8 meters / second, and motion jitter degree 0.6 m / s². Finally, these scalar features are concatenated and normalized to form a first user behavior vector. It is worth noting that if the user performs the same waving action in the main song and chorus stages of the performance, this step will generate almost identical first user behavior vectors at these two different times.

[0040] Exemplarily, in step S3, the context vector set is subjected to context attention fusion and modulation parameter generation to obtain a modulation parameter pair. It should be understood that the existing scheme for fusing multi-modal context vectors through linear weighted summation is based on a core linear hypothesis, that is, the final fused context is a linear combination of each independent context source. However, in virtual performance scenarios rich in artistic expression, different context modalities, such as the mood of the music, the narrative chapter of the performance, and the overall atmosphere of the live audience, are highly nonlinear and dynamically related in nature. Linear models cannot effectively capture the complex interaction effects between these context modalities. For example, when the music context presents a very high energy and the narrative context is in the chorus, these two pieces of information are highly synergistic and should mutually enhance each other to jointly shape an exciting comprehensive situation. Conversely, if the music is also very high in energy but the narrative is still in the prelude, this often implies a contrast of a kind of build-up, and the high energy of the music should be adjusted by the narrative stage, and its interpretation should also be different. Due to the limitations of the weight calculation, the linear weighted model is difficult to dynamically adjust the influence of one modality according to the state of another modality, and therefore has a natural defect in representing such complex and non-synergistic context combinations.

[0041] To address the aforementioned issues, a context interaction layer based on a self-attention mechanism is introduced. The core of this mechanism is that instead of evaluating the importance of each context modality in isolation, it drives deep information interaction between different context vectors, allowing them to dynamically influence each other based on their respective states, thereby generating a context representation that has undergone nonlinear deep fusion.

[0042] In one embodiment, such as Figure 5 As shown, the process of performing context attention fusion and modulation parameter generation on the context vector set to obtain modulation parameter pairs includes: S31, performing context nonlinear fusion based on self-attention mechanism on the context vector set to obtain fused context vectors; S32, inputting the fused context vectors into the parameter generation network to obtain the modulation parameter pairs.

[0043] Specifically, the context vector set is subjected to nonlinear fusion based on a self-attention mechanism to obtain a fused context vector, including: projecting the context vector set onto the query space, key space, and value space to obtain a query matrix, a key matrix, and a value matrix; performing context interaction on the query matrix, key matrix, and value matrix to obtain an interactive context matrix; and aggregating information from the interactive context matrix to obtain the fused context vector.

[0044] First, the set of context vectors is projected onto the query space, key space, and value space to obtain the query matrix, key matrix, and value matrix. Specifically, the input music context vectors, narrative context vectors, and crowd atmosphere context vectors are aggregated into a single context matrix. Subsequently, through three independent and learnable linear transformation matrices Each of these Projection yields the query matrix Q, key matrix K, and value matrix V. This assigns each context modality three distinct roles: as a queryer (Query) to actively explore connections with other modalities, as a retrieved item (Key) to represent its key characteristics, and as a value carrier (Value) to contribute its substantive content. In the context of a virtual concert, this means that musical energy is no longer static but is given dynamic interactive capabilities: it can actively query the current performance section and respond to inquiries from the crowd's atmosphere. This role separation lays the foundation for subsequent, more complex interactive information filtering and reorganization, allowing the original contextual information to engage in dialogue within a more structured space. The calculation process for this step can be expressed by the formula:

[0045]

[0046]

[0047] wherein, is the input matrix stacked by each modality context vector; and are the learnable weight matrices for generating query, key, value matrices respectively; is the query, key, value matrix after projection.

[0048] Then, attention score calculation and context interaction are performed. This process generates an attention weight distribution by calculating the dot product similarity of the query matrix Q and the key matrix K, and then scaling and Softmax nonlinear activation function. Finally, the value matrix V is weighted and aggregated with this weight. The calculation process of this step can be represented by the formula:

[0049] wherein, Q, K, V are query, key, value matrices respectively; is the transpose of the key matrix K; is the dimension of the key vector, which is used to scale the dot product result to maintain gradient stability; the softmax function converts the similarity score into an attention weight with a sum of 1.

[0050] The purpose of this step is to quantitatively and dynamically calculate the correlation strength between any two context modalities. In the scene of high-energy music with prelude, the model can calculate the similarity score between the Q of the prelude and the K of the high-energy music through learning, so that the direct influence of the music energy at the current time is naturally reduced in the generated attention weight. In this way, a post-interaction context matrix is output, each row vector of which has absorbed the correlation-weighted adjustment information from all other modalities, thereby achieving a collaborative understanding of the current comprehensive situation, enabling the system to intelligently amplify collaborative information and reconcile conflicting information.

[0051] Finally, fusion is performed to generate the final single context vector. After each modality completes information interaction and forms a new representation matrix , it is necessary to aggregate these vectors containing rich interaction information into a unified, fixed-dimension fusion vector . Specifically, average pooling aggregation operation is usually used. It should be understood that the downstream modulation parameter generation network needs a single, comprehensive vector as input to describe the overall situation at this moment. In the scene, this means summarizing the complex dialogue results between music, narrative and the crowd into a concise situation report. In this way, a fusion context vector To provide high-quality input signals for subsequent generation of delicate and appropriate sound-light interaction effects.

[0052] The technical purpose of adopting this mechanism is to overcome the limitations of linear models in context fusion tasks and construct a fusion module that can deeply understand and represent the complex nonlinear relationships between multi-modal contexts. The resulting technical effects are significant. First, it gives the system stronger context representation capabilities. The fused vector can capture advanced semantic relationships between modalities such as enhancement, inhibition, or conditional dependence, making the understanding of the scene more accurate and profound. Second, this mechanism brings higher context adaptability, allowing dynamic adjustment of the mutual influence between modalities according to real-time input and changing contexts. This enables the system to exhibit excellent robustness and intelligence when faced with various complex and even unforeseen performance situations. Ultimately, all of this translates into an improvement in the end-user experience. By providing higher-quality fused context signals to subsequent modules, the system can generate sound-light interaction effects that are more delicate, more artistically expressive, and perfectly matched to the current performance atmosphere, greatly enhancing the immersion and emotional resonance of the virtual performance scene.

[0053] Finally, the fused context vector is input into a parameter generation network to obtain the modulation parameter pair. The parameter generation network is usually a pre-trained feedforward neural network that receives the fused context vector as input and outputs a pair of numerical vectors, i.e., the final output modulation parameter pair. In a specific embodiment, the parameter generation network can adopt a standard multilayer perceptron in structure. Its function is to receive a fixed-dimensional fused context vector representing the current comprehensive situation and nonlinearly map it to a pair of modulation parameters for subsequent feature-level modulation. The input layer of this network is responsible for receiving the fused context vector, and the number of its neurons is exactly the same as the dimension of the fused context vector. The output of the input layer is provided to a series of hidden layers for processing, for example, two fully connected hidden layers can be set. Each hidden layer performs a linear transformation on the output of the previous layer and then applies a nonlinear activation function, such as a rectified linear unit, to enable the network to learn more complex relationships between context features. After hierarchical feature extraction by the hidden layers, the output is sent to a final output layer. This output layer is also a fully connected layer, but it uses a linear activation function, i.e., does not apply additional nonlinear effects to the linear transformation result, to allow the modulation parameters to have a range of any real numbers. The number of neurons in this output layer is precisely set to twice the dimension of the intermediate feature vector in the user behavior to be modulated, and can be structurally predefined for segmentation. The first half of the components in the output layer are designated as the modulation parameter γ, and the second half are designated as the modulation parameter β.

[0054] Exemplarily, in step S4, the first user behavior vector is feature-level modulated and effect parameter decoded based on the modulation parameter pair to obtain an effect parameter vector. It should be understood that in the prior art, how to effectively apply context information to the interpretation process of user behavior is a technical difficulty. This step introduces a dynamic modulation method at the feature level, so that the modulation parameter pair determined by the performance context can directly and finely adjust the deep representation of user behavior features in the neural network, thereby ensuring that the finally generated sound and light effects can simultaneously and organically reflect the user's immediate actions and the current context of the performance, which is the key to realizing dynamic adaptation of interactive mapping.

[0055] In one embodiment, as shown in Figure 6 , the feature-level modulation and effect parameter decoding of the first user behavior vector based on the modulation parameter pair to obtain an effect parameter vector comprises: S41, inputting the first user behavior vector into a backbone neural network to obtain a user behavior intermediate feature vector; S42, performing feature-level affine transformation on the user behavior intermediate feature vector based on the modulation parameter pair to obtain a modulated user behavior feature vector; S43, inputting the modulated user behavior feature vector into a decoder to obtain the effect parameter vector.

[0056] Specifically, first, the first user behavior vector is input into a backbone neural network to obtain a user behavior intermediate feature vector. The backbone neural network, for example, a 4-layer MLP structure, performs a series of nonlinear transformations on the input first user behavior vector, aiming to extract a higher-order and more abstract feature representation, which is the user behavior intermediate feature vector. Second, the user behavior intermediate feature vector is feature-level affine transformed based on the modulation parameter pair generated in the previous step to obtain a modulated user behavior feature vector. It applies two modulation parameters (denoted as and ) in the modulation parameter pair to the user behavior intermediate feature vector (denoted as ), and the calculation formula of the transformation can be represented as:

[0057] wherein, the symbol represents element-wise multiplication, and the modulated user behavior feature vector is denoted as . In this transformation, the modulation parameter is also called a multiplicative parameter vector, which scales each dimension of the intermediate feature , and the modulation parameter is also called an additive parameter vector, which translates the intermediate feature . Since the modulation parameter pairThe affine transformation is determined by the performance context itself, so it can selectively amplify, reduce or adjust the specific dimensions of the user behavior features according to the current context, so as to obtain a new feature vector that has been effectively modulated by the context information, i.e. the modulated user behavior feature vector.

[0058] Finally, the modulated user behavior feature vector is input into the decoder to obtain the effect parameter vector. The function of the decoder is to map and decode the modulated user behavior feature vector containing the fusion information of user behavior and context into an effect parameter vector containing specific sound and light effect parameters in a fixed format. In a specific embodiment, the decoder can adopt a multi-layer perception in structure, and its core function is to map and convert a high-order abstract modulated user behavior feature vector that has fused user behavior and context information into a structured effect parameter vector containing specific sound and light effect parameters. The input layer of the network is configured to receive the modulated user behavior feature vector of the previous stage. After the input layer, the decoder can contain two fully connected hidden layers for hierarchical feature decoding. The first hidden layer receives the modulated user behavior feature vector, performs linear transformation thereon, and applies a rectified linear unit (ReLU) activation function for nonlinear processing. The output of this layer is then sent to the second hidden layer, which also performs a fully connected linear transformation and ReLU activation. Through such layer-by-layer dimension reduction and nonlinear mapping, the network can gradually decode the abstract fusion features into intermediate representations more relevant to the final effect parameters. After processing by the hidden layers, the output vector is sent to a final output layer. The structural design of the output layer is the key of the decoder, and the number of neurons thereof is accurately set to the total dimension of the effect parameter vector, and different activation functions are used for different types of parameters. In a specific embodiment, if the effect parameter vector contains 8 logical values representing special effect types, 3 values representing RGB colors, 1 value representing size, 1 value representing life cycle, and 3 values representing initial speed direction, the total number of neurons of the output layer is 16. For the 8 components in the output vector representing special effect types, a linear activation function is used to output the original logical values; for the 3 components representing RGB colors, a Sigmoid activation function is used to ensure that the output values are constrained within the effective range of 0 to 1; for the 2 components representing size and life cycle, a ReLU activation function is used to ensure that the output values are non-negative; and for the 3 components representing initial speed direction, a linear activation function is also used to allow them to take any positive or negative real values. Through such output layer design with mixed activation functions, the decoder finally outputs an effect parameter vector with a dimension of 16.

[0059] Exemplarily, in step S5, the effect parameter vector is vector-decoded and rendered-instruction-converted to obtain the rendering instruction. It should be appreciated that the effect parameter vector itself is a set of floating-point numbers without direct executable information and cannot be directly used by the rendering pipeline. Therefore, the intention of the algorithm, i.e., the parameters contained in the effect parameter vector, needs to be accurately translated into the final visual and auditory presentation through vector-decoding and rendered-instruction-conversion.

[0060] In one embodiment, the effect parameter vector is vector-decoded and rendered-instruction-converted to obtain the rendering instruction, including: vector-decoding and semantically structuring the effect parameter vector to obtain a semanticized effect descriptor; and rendering-resource-mapping and instruction-parameterizing the semanticized effect descriptor to obtain the rendering instruction.

[0061] Specifically, first, the effect parameter vector is vector-decoded and semantically structured to obtain a semanticized effect descriptor. This process first parses the input effect parameter vector according to a predefined parameter mode, corresponding the numerical values in different index ranges in the vector to specific semantic meanings (such as color, size, special effect type, etc.). Subsequently, the parsed numerical components are subjected to necessary conversion and inverse normalization processing, for example, decoding logical values representing categories into specific category labels through argmax operation, restoring continuous values in the normalized interval to their physical value range through linear mapping, and converting color values in a specific format (such as HSV) to a color space commonly used by rendering engines (such as RGB). After conversion, all these parameters with explicit semantics are encapsulated into a structured data object, i.e., a semanticized effect descriptor.

[0062] Then, the semanticized effect descriptor is rendered-resource-mapped and instruction-parameterized to obtain the rendering instruction. This process first extracts the special effect type field from the semanticized effect descriptor and uses it as an index to find and match a specific rendering asset template (such as a particle system prefab) in the preloaded special effect asset library. Then, an instantiation instruction is generated to create an instance of the asset template at the user-specified location (usually derived from the user's real-world coordinates). Finally, all the remaining parameters in the semanticized effect descriptor are traversed, and a parameter setting instruction is generated for each parameter to bind the parameter value to the corresponding attribute of the newly created special effect instance. All these instantiation and parameter setting instructions collectively constitute the rendering instruction as the final output of this step.

[0063] In summary, the virtual reality-based stage sound and light interaction method provided in the application has been illustrated. The method first acquires two types of input data in parallel: one is to comprehensively quantify the context information of the performance environment, specifically by processing real-time audio stream, performance timeline data and historical user data, extracting the core features thereof respectively, and vectorizing to obtain a context vector set; the other is to accurately represent the specific behavior of the user, by processing the raw sensor data stream of the VR device worn by the user, extracting the kinematic features thereof, thereby obtaining a user behavior vector. Then, a dynamic modulation link is introduced. The link first uses a context attention fusion mechanism to process the multi-modal context vector set to generate a pair of modulation parameters that can reflect the current comprehensive situation. Subsequently, the user behavior vector is input into a backbone network for feature extraction, and the modulation parameters generated by the foregoing are used to perform a feature-level modulation transformation on the intermediate features in the network. After this step, the original user behavior features are effectively injected with the current context information. Finally, the modulated feature vector is decoded by a decoder to generate an effect parameter vector, which is parsed and converted to form the final rendering instructions executable by the rendering engine. In this way, the sound and light feedback to the user's behavior can be dynamically adjusted according to the performance context, achieving accurate matching of the interaction effect and the artistic atmosphere of the stage, and significantly improving the artistic expressiveness of the user's immersion and interaction experience.

[0064] The application also provides a virtual reality-based stage sound and light interaction system for executing the virtual reality-based stage sound and light interaction method described above, as shown in Figure 7 The virtual reality-based stage sound and light interaction system 700 includes a multi-modal context encoding module 710 for multi-modal context vectorization of the acquired real-time audio stream, performance timeline data and historical user data to obtain a context vector set; a user behavior feature extraction module 720 for user behavior feature extraction of the raw sensor data stream of the first user to obtain a first user behavior vector; a context attention modulation module 730 for context attention fusion and modulation parameter generation of the context vector set to obtain a modulation parameter pair; a feature modulation and effect decoding module 740 for feature-level modulation and effect parameter decoding of the first user behavior vector based on the modulation parameter pair to obtain an effect parameter vector; and a rendering instruction generation module 750 for vector parsing and rendering instruction conversion of the effect parameter vector to obtain a rendering instruction.

[0065] The computer readable storage medium provided in the embodiments of the application has computer program codes stored therein, which, when executed on a computer, causes the computer to perform the related method steps to implement the virtual reality-based stage sound and light interaction method provided in the above embodiments.

[0066] The embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to execute the above-mentioned related steps to realize the virtual reality-based stage sound and light interaction method provided by the above-mentioned embodiment.

[0067] Wherein, the system, computer readable storage medium or computer program product provided by the embodiment of the present application are all used to execute the corresponding method provided above, thus the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, which will not be described here.

[0068] It should be noted that the above-mentioned sequence of the embodiments is only for description, and does not represent the advantages and disadvantages of the embodiments.

[0069] The processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing can be advantageous. The various embodiments described herein are described using a progression of embodiments, additional implementations can be obtained from a combination of features of the described embodiments.

Claims

1. A stage sound and light interaction method based on virtual reality, characterized in that, include: The acquired real-time audio stream, performance timeline data, and historical user data are multimodal context vectorized to obtain a set of context vectors; User behavior features are extracted from the raw sensor data stream of the first user to obtain the first user behavior vector; Context attention fusion and modulation parameter generation are performed on the context vector set to obtain modulation parameter pairs; Based on the modulation parameter pair, feature-level modulation and effect parameter decoding are performed on the first user behavior vector to obtain the effect parameter vector; The effect parameter vector is parsed and converted into rendering instructions to obtain the rendering instructions.

2. The stage sound and light interaction method based on virtual reality according to claim 1, characterized in that, The acquired real-time audio stream, performance timeline data, and historical user data are multimodal context vectorized to obtain a set of context vectors, including: Music feature extraction is performed on real-time audio streams to obtain music context vectors; The performance timeline data is encoded with both real and virtual states to obtain the narrative context vector; Extract crowd atmosphere features from historical user data to obtain crowd atmosphere context vectors; The music context vector, narrative context vector, and crowd atmosphere context vector are packaged together to obtain a context vector set.

3. The stage sound and light interaction method based on virtual reality according to claim 2, characterized in that, The performance timeline data is encoded using both real and virtual states to obtain a narrative context vector, including: Determine the current performance chapter based on the current timestamp's position in the performance timeline data; One-hot encoding is performed on the current performance chapter to obtain the narrative context vector.

4. The stage sound and light interaction method based on virtual reality according to claim 1, characterized in that, User behavior features are extracted from the raw sensor data stream of the first user to obtain the first user behavior vector, including: The kinematic basic features of the raw sensor data stream of the first user are calculated in real time to obtain the kinematic feature time series. Each frame of data in the raw sensor data stream is 6DoF attitude data. Spatiotemporal high-order feature aggregation is performed on the kinematic feature time series to obtain an aggregated scalar feature set; The aggregated scalar feature set is vectorized and normalized to obtain the first user behavior vector.

5. The stage sound and light interaction method based on virtual reality according to claim 4, characterized in that, Spatiotemporal high-order feature aggregation is performed on the kinematic feature time series to obtain an aggregated scalar feature set, including: The length of the hand trajectory is obtained by calculating the trajectory geometric features of the kinematic time series. Spatial occupancy features are calculated from the kinematic time series to obtain the spatial volume of hand movements; Kinematic feature time series are subjected to motion statistical feature calculation to obtain average velocity scalar and motion jitter, wherein the hand trajectory length, the volume of the two hands' activity space, the average velocity scalar and motion jitter constitute the aggregated scalar feature set.

6. The stage sound and light interaction method based on virtual reality according to claim 1, characterized in that, Context attention fusion and modulation parameter generation are performed on the context vector set to obtain modulation parameter pairs, including: A self-attention mechanism-based nonlinear fusion of context vectors is performed on the context vector set to obtain a fused context vector; The fusion context vector input parameters are used to generate a network to obtain the modulation parameter pair.

7. The stage sound and light interaction method based on virtual reality according to claim 6, characterized in that, A self-attention-based nonlinear fusion of context vectors is performed on the context vector set to obtain a fused context vector, including: Project the set of context vectors onto the query space, key space, and value space to obtain the query matrix, key matrix, and value matrix; Perform context interaction on the query matrix, key matrix, and value matrix to obtain the interactive context matrix; Information is aggregated from the context matrix after interaction to obtain a fused context vector.

8. The stage sound and light interaction method based on virtual reality according to claim 1, characterized in that, Based on the modulation parameter pair, feature-level modulation and effect parameter decoding are performed on the first user behavior vector to obtain the effect parameter vector, including: The first user behavior vector is input into the backbone neural network to obtain the intermediate feature vector of user behavior. Based on the modulation parameter pair, a feature-level affine transformation is performed on the intermediate feature vector of user behavior to obtain the modulated user behavior feature vector. The modulated user behavior feature vector is input into the decoder to obtain the effect parameter vector.

9. The stage sound and light interaction method based on virtual reality according to claim 1, characterized in that, The effect parameter vector is parsed and converted into rendering instructions to obtain rendering instructions, including: The effect parameter vector is vector-decoded and semantically structured to obtain a semantic effect descriptor; The semantic effect descriptor is rendered by mapping rendering resources and parameterizing instructions to obtain the rendering instructions.

10. A stage audio-visual interactive system based on virtual reality, characterized in that, include: The multimodal context encoding module is used to perform multimodal context vectorization on the acquired real-time audio stream, performance timeline data and historical user data to obtain a set of context vectors; The user behavior feature extraction module is used to extract user behavior features from the raw sensor data stream of the first user to obtain the first user behavior vector. The context attention modulation module is used to perform context attention fusion and modulation parameter generation on the context vector set to obtain modulation parameter pairs; The feature modulation and effect decoding module is used to perform feature-level modulation and effect parameter decoding on the first user behavior vector based on the modulation parameter pair to obtain the effect parameter vector; The rendering instruction generation module is used to perform vector parsing and rendering instruction conversion on the effect parameter vector to obtain the rendering instruction.

Images