Digital performing art particle special effect generation method and system based on multi-modal motion capture

Abstract

The application belongs to the technical field of computers, and particularly relates to a digital performing art particle special effect generation method and system based on multi-modal motion capture, which comprises five steps of multi-modal data synchronous acquisition, physiological signal extraction and analysis, emotional state fusion determination, differentiated special effect generation and real-time rendering output, and comprises five modules of multi-modal perception hardware, physiological signal analysis, emotional state determination, special effect parameter generation and real-time rendering engine. The application synchronously acquires kinematic data and physiological state data, adopts a double-flow space-time graph convolution network to fuse action features and physiological features to realize accurate emotion recognition, establishes a mapping rule library of emotional states to special effect parameters to dynamically generate differentiated particle special effect parameters, can realize differentiated generation of special effects under different emotional states for the same action type, and significantly improves the expressiveness and emotional narrative ability of a digital performing art scene.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically a method and system for generating digital performance particle effects based on multimodal motion capture. Background Technology

[0002] The development of digital performance technology continues to drive content innovation in scenarios such as virtual idol performances, AR / VR immersive performances, and digital live streaming. Particle effects, as a core element of visual presentation, play a key role in enhancing the appeal of performances and conveying emotional information. The accuracy of matching special effects with the emotional semantics of performances directly determines the expressiveness of digital performance content and the audience's immersive experience.

[0003] Existing motion-driven particle effects generation technologies mostly rely on multimodal motion capture solutions. These solutions use sensor devices to collect kinematic parameters such as the spatial position, motion trajectory, and angular velocity of human skeletal joints. Based on preset effects templates, they establish a mapping relationship between motion features and visual effects, enabling basic linkage generation of motion and effects in conventional standardized performance scenarios.

[0004] However, these technologies generally suffer from the inherent flaw of having a single dimension of motion information. The capture paradigm based on pure kinematic data cannot perceive the actor's breathing rhythm, muscle tension, and other internal physiological states. Physiological states, as the internal driving force for motion generation, directly determine the emotional semantic differences of the same external action in different performance contexts. The lack of physiological perception dimension means that particle effects generation can only perform template matching based on external action features, and cannot identify the differentiated emotional states contained in the same type of action. This results in a semantic break between the effects and the performance intention. The audience can only receive the surface-level linkage between the actions and effects, lacking the coherence of emotional narrative, which significantly restricts the content expressiveness and immersiveness of digital performance scenes.

[0005] Therefore, this invention provides a method and system for generating digital performance particle effects based on multimodal motion capture. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0007] The technical solution adopted by this invention to solve its technical problem is: the digital performance particle effects generation method based on multimodal motion capture described in this invention includes the following steps: Step 1: Multimodal motion data synchronous acquisition. The system synchronously acquires the actor's kinematic and physiological data through a multimodal sensing array deployed in the performance area. The kinematic data includes the three-dimensional spatial position, motion trajectory, and angular velocity parameters of skeletal joints. The physiological data includes respiratory airflow pressure signals, muscle surface electromyography signals, and center of gravity displacement signals. The multimodal sensing array includes an optical motion capture subsystem, a respiratory airflow sensing subsystem, an electromyography signal acquisition subsystem, and an inertial measurement unit. The sampling clocks of all subsystems are aligned at the millisecond level through a unified clock synchronization protocol. After data fusion preprocessing, the acquired raw data generates a multidimensional feature vector sequence containing spatiotemporal stamps, providing a complete data source with temporal consistency for subsequent physiological state analysis.

[0008] Step Two: Physiological State Signal Extraction and Analysis. This invention performs feature extraction and state analysis on the raw physiological signals acquired by the multimodal sensing array. This step uses an independent component analysis algorithm to separate the feature components related to muscle tension from the electromyography signal. An adaptive filtering algorithm is used to extract respiratory frequency and respiratory depth parameters from the respiratory airflow pressure signal. A wavelet transform algorithm is used to decompose the center of gravity displacement signal into a multi-scale value to obtain the center of gravity stability index. The wavelet transform uses the db4 wavelet basis, and the decomposition level is set to 5 levels. After the analysis process performs feature extraction independently for each physiological signal, the extracted respiratory rhythm features, muscle tension features, and center of gravity stability features are fused at the feature level using a multi-signal fusion framework to generate a physiological feature vector that reflects the actor's internal physiological state. This vector accurately represents the intensity of the actor's physiological driving force during the execution of the action, providing an objective physiological basis for the determination of emotional state.

[0009] Step 3: Emotional State Fusion and Judgment Step. This invention deeply fuses the physiological feature vectors output from the physiological state signal extraction step with the action feature parameters extracted from kinematic data to construct an emotional state judgment model. This step adopts a dual-stream spatiotemporal graph convolutional network architecture. The dual-stream spatiotemporal graph convolutional network is trained using the cross-entropy loss function, and the optimizer uses the Adam algorithm. The first stream processes the spatial topological relationships and temporal series changes of skeletal joints to extract action type features, while the second stream processes the nonlinear mapping of physiological feature vectors to extract physiological driving features. The dual-stream network dynamically weights the contribution ratio of the two features through an attention mechanism at the feature fusion layer, outputting emotional state classification results, specifically including four differentiated emotional states: calm gaze, enthusiastic exuberance, somber farewell, and hesitant hesitation. The judgment results accurately reflect the differences in emotional semantics carried by the same action type under different physiological drives, providing precise semantic labels at the emotional level for the differentiated generation of particle effects.

[0010] Step Four: Differentiated Particle Effect Generation Step. Based on the emotional state classification results output by the emotional state fusion judgment step, this invention dynamically selects and adjusts the particle effect generation parameters. This step constructs a mapping rule library from emotional states to particle effect parameters. Each emotional state corresponds to a set of effect generation parameters, including particle shape parameters, light effect color parameters, motion trajectory parameters, and lifespan parameters. For example, for the calm gaze emotional state, the corresponding particle shape parameters are configured as a soft halo shape, the light effect color parameters as a warm gradient spectrum, the motion trajectory parameters as a slowly rising floating trajectory, and the lifespan parameter as a long decay duration. For the somber farewell emotional state, the corresponding particle shape parameters are configured as a dim dissipation shape, the light effect color parameters as a cool decay spectrum, the motion trajectory parameters as a slowly sinking dissipation trajectory, and the lifespan parameter as a rapid decay duration. The parameter configurations in the mapping rule library are optimized and stored through offline training. The system dynamically looks up the corresponding effect generation parameters based on the real-time determined emotional state, eliminating the need to preset independent effect templates for each emotional state, thus achieving differentiated generation of effects for the same action type under different emotional states.

[0011] Step 5: Real-time rendering output. Based on the special effects generation parameters obtained from the differentiated particle effects generation step, the system drives the particle engine to perform real-time special effects rendering. This step simulates the generation, movement, collision, and destruction of particles based on a physically based particle system. The rendering pipeline adopts a deferred shading architecture to support real-time light effect calculations for a large number of particles. The rendered video stream is pushed to the terminal display device in real time through the content delivery network of the digital performing arts platform, ensuring that the audience can perceive particle effects that are accurately matched with the actors' movements and emotional states under low latency conditions.

[0012] Preferably, in the multimodal motion data synchronous acquisition step, the optical motion capture subsystem includes at least 8 infrared motion capture cameras with a sampling rate of 240 frames / second and a spatial positioning accuracy of sub-millimeter level; the respiratory airflow sensing subsystem uses a thin-film pressure sensor array that is attached to the actor's chest and abdomen breathing area; the electromyography signal acquisition subsystem uses surface electrode patches, and the acquisition position covers the muscle belly areas of the pectoralis major, latissimus dorsi, and rectus abdominis muscles; and the inertial measurement unit integrates a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer with a sampling rate of 1000 Hz per second.

[0013] Preferably, in the physiological state signal extraction and analysis steps, the preprocessing of electromyography (EMG) signals includes bandpass filtering with a passband range of 20 Hz to 500 Hz to remove baseline drift and environmental electromagnetic interference. Feature extraction uses time-domain statistical analysis to calculate the root mean square (RMS) and integral EMG values ​​of the EMG signals as quantitative indicators of muscle tension. The preprocessing of respiratory airflow pressure signals includes low-pass filtering with a cutoff frequency of 5 Hz to extract low-frequency components related to breathing. Feature extraction uses a peak detection algorithm to calculate the inspiratory duration, expiratory duration, and respiratory interval within the respiratory cycle to characterize respiratory rhythm features. The processing of the center of gravity displacement signal includes converting the triaxial acceleration signal into a center of gravity displacement angle and evaluating the degree of center of gravity stability by calculating the standard deviation.

[0014] Preferably, in the emotional state fusion determination step, the first-stream input of the dual-stream spatiotemporal graph convolutional network is a three-dimensional coordinate sequence of skeletal joints, constructing a spatiotemporal graph structure with joints as nodes and skeletal lines as edges. The graph convolution operation uses multi-scale spatial convolution kernels to extract spatial relationship features between joints, and the temporal dimension uses temporal convolution kernels to extract the temporal evolution features of joint movement. The second-stream input is a physiological feature vector, which is nonlinearly mapped through a multilayer perceptron to obtain physiological driving features. The feature fusion layer uses a soft attention mechanism to calculate the attention weights of the first-stream and second-stream features. The fusion formula is a weighted summation, and the weights are obtained through end-to-end learning of the network.

[0015] Preferably, in the differentiated particle effects generation step, the mapping rule base from emotional state to particle effect parameters is established through an offline training phase. The training data includes a large number of actor action samples and their corresponding expert-annotated emotional tags and effect parameter configurations. The training process uses a reinforcement learning algorithm to optimize the effect parameters to maximize the audience's emotional resonance score. The completed mapping rule base includes at least four emotional states and their corresponding effect parameter sets. The effect parameter set for each emotional state includes the particle quantity range, particle size range, color gradient curve, motion speed range, and lifespan range.

[0016] Preferably, in the real-time rendering output step, the particle engine adopts a parallel particle simulation architecture based on computation shaders, which supports real-time computation of more than 500,000 particles simultaneously. The rendering pipeline supports screen space ambient occlusion and Bloom effects to enhance the visual expressiveness of particle effects. The output resolution supports the 4K ultra-high-definition standard, the frame rate is stable at 60 frames per second, and the rendering latency is less than 16 milliseconds.

[0017] The present invention also provides a digital performance particle effects generation system based on multimodal motion capture, the system comprising a multimodal perception hardware module, a physiological signal analysis module, an emotional state determination module, an effects parameter generation module, and a real-time rendering engine module.

[0018] The multimodal perception hardware module is configured to construct a multi-dimensional data acquisition matrix in the performance area. This module includes an optical motion capture unit, a respiratory airflow sensing unit, an electromyography signal acquisition unit, and an inertial measurement unit. All units are electrically coupled to the system control center through a high-speed data bus. The sampling clock is time-aligned through a unified clock synchronization protocol to ensure precise time alignment between kinematic data and physiological state data. The perception hardware module not only collects motion trajectory information of skeletal joints, but also collects respiratory, electromyography, and center of gravity signals that reflect the actor's internal physiological driving state, providing multi-dimensional and complete information input for emotional state determination.

[0019] The physiological signal analysis module is electrically connected to the multimodal perception hardware module. It integrates a multi-channel signal processing unit, which is responsible for receiving raw physiological signals and performing feature extraction and state analysis. The electromyography signal processing unit separates muscle tension-related feature components through independent component analysis algorithm. The respiratory signal processing unit extracts respiratory frequency and respiratory depth parameters through adaptive filtering algorithm. The center of gravity signal processing unit evaluates the stability of the center of gravity through wavelet transform algorithm. This module integrates the features of each channel through a multi-signal fusion framework to output physiological feature vectors, providing objective quantitative basis at the physiological level for emotional state determination.

[0020] The emotional state determination module is connected to the physiological signal analysis module and is configured to perform a deep fusion determination of action features and physiological features. The module integrates a dual-stream spatiotemporal graph convolutional network processor. The first stream processor constructs a skeletal spatiotemporal graph structure and performs graph convolution operations to extract action type features. The second stream processor performs nonlinear mapping on the physiological feature vector to extract physiological driving features. The feature fusion layer dynamically weights the contribution ratio of the two features through an attention mechanism and outputs the emotional state classification result. This module can identify the differentiated emotional semantics carried by the same action type under different physiological drives, and achieve accurate understanding of performance intentions.

[0021] The special effects parameter generation module is connected to the emotion state determination module and is configured to dynamically generate particle special effects parameters based on the emotion state classification results. This module integrates a mapping rule library memory to store the correspondence between various emotion states and special effects parameter sets. When the emotion state determination module outputs an emotion state label, the special effects parameter generation module automatically looks up the table to obtain the corresponding special effects generation parameter set, including particle shape parameters, light effect color parameters, motion trajectory parameters, and life cycle parameters. This module realizes the dynamic association between special effects parameter generation and emotion state, and can generate differentiated visual effects without the need to preset independent special effects templates.

[0022] The real-time rendering engine module integrates a computational shader-based parallel particle simulator and a deferred shading renderer. This module receives special effects generation parameters output by the special effects parameter generation module, drives the particle system to perform real-time rendering calculations, and pushes the rendering output to the terminal display device in real time through the content distribution network of the digital performing arts platform. This module ensures that the particle special effects are precisely synchronized with the actors' actions and emotional states, so that the audience can perceive a coherent emotional narrative visual effect under low latency conditions.

[0023] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention discloses a method and system for generating digital performance particle effects based on multimodal motion capture. By introducing the acquisition dimension of physiological state signals through the synchronous acquisition step of multimodal motion data, a complete multimodal dataset covering kinematic data and physiological state data is constructed. This dataset not only includes external kinematic parameters such as the spatial position, motion trajectory and angular velocity of skeletal joints, but also introduces key parameters reflecting the internal physiological driving state, such as respiratory airflow pressure, muscle surface electromyography and center of gravity displacement. This multi-dimensional information fusion architecture breaks through the paradigm limitation of existing technologies that rely solely on kinematic data, and lays an information foundation for the emotional generation of particle effects.

[0024] 2. The present invention discloses a digital performing arts particle effects generation method and system based on multimodal motion capture. Through the physiological state signal extraction and analysis step, the method achieves accurate quantitative representation of the actor's internal physiological state. This step extracts physiological feature parameters such as breathing rhythm, muscle tension, and center of gravity stability from multimodal raw data through signal processing algorithms such as independent component analysis, adaptive filtering, and wavelet transform. The method then uses a multi-signal fusion framework to generate a physiological feature vector. This vector accurately reflects the intensity of the actor's physiological driving force during the execution of the action, enabling the system to understand the emotional semantic differences of the same external action under different internal physiological states.

[0025] 3. The present invention provides a method and system for generating digital performance particle effects based on multimodal motion capture. Through an emotional state fusion judgment step, it achieves accurate identification of the emotional intention of the performance. This step uses a dual-stream spatiotemporal graph convolutional network architecture to deeply fuse action features and physiological features. By dynamically weighting the contribution ratio of the two features through an attention mechanism, it can distinguish various differentiated emotional states such as calm gaze, passionate exuberance, solemn farewell, and hesitation. This technical feature solves the technical deficiency of existing technologies that cannot understand action intentions and emotional states, and achieves accurate semantic matching between particle effects and performance emotions.

[0026] 4. The present invention provides a method and system for generating particle effects in digital performances based on multimodal motion capture. Through differentiated particle effect generation steps and real-time rendering output steps, it achieves differentiated generation of effects for the same action type under different emotional states. This step achieves dynamic lookup configuration of effect parameters through a mapping rule library from emotional state to particle effect parameters. It automatically selects the corresponding particle shape, light effect color, motion trajectory, and life cycle parameters for different emotional states. Differentiated visual effects can be generated without the need to preset independent effect templates. This technical feature breaks through the limitation of existing technologies that can only generate one effect template for the same action, and significantly improves the expressiveness and emotional narrative ability of digital performance scenes. Attached Figure Description

[0027] The invention will now be further described with reference to the accompanying drawings.

[0028] Figure 1 This is a structural block diagram of a digital performing arts particle effects generation system based on multimodal motion capture, as described in this invention. Figure 2 This is a flowchart of a method for generating digital performance particle effects based on multimodal motion capture in this invention. Detailed Implementation

[0029] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0030] Example 1: like Figure 1 As shown, this embodiment provides a digital performance particle effects generation system based on multimodal motion capture, which is applied to the real-time particle effects generation in digital performance scenes. It is used to collect the kinematic data and physiological state data of actors in real time during the performance, generate accurate particle effects parameters by judging the emotional state, and drive the real-time rendering engine to output particle effects visual effects that are accurately matched with the actor's movements and emotional state.

[0031] The system comprises five core functional modules: a multimodal perception hardware module, a physiological signal analysis module, an emotional state determination module, a special effects parameter generation module, and a real-time rendering engine module. These modules are electrically connected and transmit data through a high-speed data bus.

[0032] The multimodal sensing hardware module is set up around the performance stage in the performance area to build a multi-dimensional data acquisition matrix. The module includes four sub-units: an optical motion capture unit, a respiratory airflow sensing unit, an electromyography signal acquisition unit, and an inertial measurement unit. All sub-units are electrically coupled to the system control center through a high-speed data bus. The sampling clock is time-aligned through a unified clock synchronization protocol to ensure accurate time alignment between kinematic data and physiological state data.

[0033] As a further implementation method, the specific deployment and connection of each sub-unit are as follows: The optical motion capture unit consists of at least 8 infrared motion capture cameras, which are deployed around the stage in the performance area according to a preset spatial layout. Each camera communicates with the system control center via a gigabit Ethernet interface. The respiratory airflow sensing unit adopts a thin-film pressure sensor array, which consists of multiple distributed pressure sensor nodes. Each sensor node is connected in series with a flexible ribbon cable and then connected to the signal conditioning circuit module. The electromyography signal acquisition unit adopts surface electrode patches, which are attached to the actor's skin surface with medical-grade conductive gel. The electrode patches are connected to the electromyography signal amplifier module via shielded cables. The inertial measurement unit is integrated into the inertial measurement component worn by the actor and communicates with the system control center via a Bluetooth wireless interface.

[0034] The optical motion capture unit comprises at least eight infrared motion capture cameras with a sampling rate of 240 frames per second and sub-millimeter spatial positioning accuracy. It is used to capture the three-dimensional spatial position, motion trajectory, and angular velocity parameters of the actor's skeletal joints during performance. The respiratory airflow sensing unit employs a thin-film pressure sensor array, fitted to the actor's chest and abdomen during respiratory fluctuations, to collect airflow pressure signals generated by the movement of the chest and abdomen during breathing. The electromyography (EMG) signal acquisition unit uses surface electrode patches covering the muscle bellies of the pectoralis major, latissimus dorsi, and rectus abdominis muscles to collect surface EMG signals generated during muscle contraction. The inertial measurement unit integrates a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, with a sampling rate of 1000 Hz. It is used to acquire inertial measurement data of the actor's body posture and calculate the center of gravity displacement signal.

[0035] The physiological signal analysis module is electrically connected to the multimodal sensing hardware module, receiving the raw physiological signals output by the multimodal sensing hardware module and performing feature extraction and state analysis. This module integrates a multi-channel signal processing unit, including three sub-units: an electromyography (EMG) signal processing unit, a respiratory signal processing unit, and a center-of-gravity (COG) signal processing unit. The EMG signal processing unit separates muscle tension-related feature components using an independent component analysis algorithm and calculates the root mean square (RMS) and integral EMG values ​​as quantitative indicators of muscle tension using time-domain statistical analysis. The respiratory signal processing unit extracts respiratory frequency and respiratory depth parameters from the respiratory airflow pressure signal using an adaptive filtering algorithm and calculates the inspiratory duration, expiratory duration, and respiratory interval within the respiratory cycle using a peak detection algorithm to characterize respiratory rhythm features. The COG signal processing unit performs multi-scale decomposition of the COG displacement signal using a wavelet transform algorithm to obtain a COG stability index and evaluates the degree of COG stability through standard deviation calculation. The physiological signal analysis module integrates the features of each channel through a multi-signal fusion framework to output a physiological feature vector, providing objective quantitative evidence at the physiological level for emotional state determination.

[0036] The emotional state determination module, connected to the physiological signal analysis module and the kinematic data processing module, is configured to perform a deep fusion determination of action features and physiological features. This module integrates a dual-stream spatiotemporal graph convolutional network processor, including a first-stream processor and a second-stream processor. The first-stream processor constructs a spatiotemporal graph structure of the three-dimensional coordinate sequence of skeletal joints. Using joints as nodes and skeletal lines as edges, it employs multi-scale spatial convolutional kernels to extract spatial relationship features between joints and temporal convolutional kernels to extract the temporal evolution features of joint movement, thereby extracting action type features. The second-stream processor receives physiological feature vectors and performs nonlinear mapping through a multilayer perceptron to extract physiological driving features. The feature fusion layer uses a soft attention mechanism to calculate the attention weights of the first-stream and second-stream features, fuses the two features through a weighted summation method, and outputs the emotional state classification result. This module can identify the differentiated emotional semantics carried by the same action type under different physiological drives, achieving a precise understanding of performance intentions.

[0037] The special effects parameter generation module is connected to the emotion state determination module and is configured to dynamically generate particle special effects parameters based on the emotion state classification results. This module integrates a mapping rule library memory, storing the correspondence between various emotion states and sets of special effects parameters. When the emotion state determination module outputs an emotion state label, the special effects parameter generation module automatically looks up the corresponding set of special effects generation parameters, including particle shape parameters, light effect color parameters, motion trajectory parameters, and lifespan parameters. Particle shape parameters define the particle's appearance shape and halo effect configuration; light effect color parameters define the particle's color configuration and gradient curve; motion trajectory parameters define the particle's movement mode and speed configuration; and lifespan parameters define the particle's duration and decay configuration. The special effects parameter generation module achieves a dynamic association between special effects parameter generation and emotion state, generating differentiated visual effects without the need for preset independent special effects templates.

[0038] The real-time rendering engine module integrates a computational shader-based parallel particle simulator and a deferred shading renderer. This module receives effects generation parameters from the effects parameter generation module and drives the particle engine to perform real-time rendering calculations. The particle engine adopts a computational shader-based parallel particle simulation architecture, supporting the simultaneous processing of over 500,000 particles in real-time. The rendering pipeline supports screen-space ambient occlusion and bloom effects to enhance the visual expressiveness of particle effects. The rendered output is pushed to the terminal display device in real time through the content delivery network of the digital performing arts platform, ensuring that the audience perceives a coherent emotional narrative visual effect with low latency.

[0039] like Figure 2 As shown, the digital performance particle effects generation method based on multimodal motion capture corresponding to this embodiment includes the following steps: Step 1: Multimodal Data Acquisition and Preprocessing. A multimodal sensing array deployed in the performance area synchronously acquires the actors' kinematic and physiological data. Kinematic data includes the three-dimensional spatial position, motion trajectory, and angular velocity parameters of skeletal joints. Physiological data includes respiratory airflow pressure signals, muscle surface electromyography signals, and center of gravity displacement signals. The sampling clocks of all sub-units are time-aligned using a unified clock synchronization protocol. The acquired raw data undergoes data fusion preprocessing to generate a multidimensional feature vector sequence containing spatiotemporal stamps.

[0040] The multimodal perception hardware module first triggers the optical motion capture unit to start working. Eight infrared motion capture cameras in the optical motion capture unit simultaneously acquire infrared images of the performance area. The cameras capture two-dimensional images of infrared reflection markers on the actor's body surface at a sampling rate of 240 frames per second. After receiving the two-dimensional image data acquired by all the infrared motion capture cameras, the system control center executes a three-dimensional reconstruction algorithm to calculate the three-dimensional spatial coordinates of each marker. The three-dimensional reconstruction algorithm uses the principle of multi-camera triangulation, utilizing the two-dimensional coordinate positions of the same marker on the imaging plane of at least two cameras, combined with the intrinsic and extrinsic parameter matrices of each camera, to calculate the three-dimensional spatial coordinates of the marker. The system control center maps the three-dimensional coordinates of all markers onto a predefined sequence of skeletal joints according to the skeletal model definition, obtaining a sequence of three-dimensional spatial positions of the skeletal joints. The system control center performs differential calculations on the three-dimensional spatial positions of the skeletal joints in adjacent frames to obtain the motion trajectory, and performs differential calculations on the direction vectors of the bone connections to obtain the angular velocity parameters.

[0041] The respiratory airflow sensing unit begins acquiring respiratory airflow pressure signals. A thin-film pressure sensor array is attached to the actor's chest and abdomen, which fluctuates with breathing. The sensor array detects pressure changes on the skin surface of the chest and abdomen in real time. When the actor inhales, the chest and abdomen rise, causing the pressure to increase; when the actor exhales, the chest and abdomen contract, causing the pressure to decrease. The sensor array converts the pressure changes into analog voltage signals. The output signal of the sensor array is sampled by an analog-to-digital converter to obtain a discrete respiratory airflow pressure signal sequence.

[0042] The electromyography (EMG) signal acquisition unit begins acquiring EMG signals from the muscle surface. Surface electrode patches are attached to the muscle belly regions of the actor's pectoralis major, latissimus dorsi, and rectus abdominis muscles. The electrode patches detect the weak electrical signals generated during muscle fiber depolarization. The acquired raw EMG signals are amplified by a preamplifier to output an analog voltage signal. This signal is then sampled by an analog-to-digital converter to obtain a discrete EMG signal sequence.

[0043] The inertial measurement unit (IMU) begins acquiring inertial measurement data. The IMU integrates a triaxial accelerometer, a triaxial gyroscope, and a triaxial magnetometer, outputting triaxial acceleration data, triaxial angular velocity data, and triaxial magnetic field strength data at a sampling rate of 1000 Hz. The system control center integrates the triaxial acceleration signals to obtain triaxial velocity signals, integrates them again to obtain triaxial displacement signals, and calculates the center of gravity displacement signal from the triaxial displacement signals.

[0044] The sampling clocks of all sub-units are time-aligned using a unified clock synchronization protocol. This embodiment employs the IEEE 1588 Precision Time Protocol (PTP), achieving sub-microsecond timing synchronization through master-slave clock timestamp interaction, ensuring that the time deviation of data from each subsystem is less than 1 millisecond. The unified clock synchronization protocol assigns a uniform clock reference time to each sub-unit, and all sub-units tag their collected raw data with a spatiotemporal stamp according to this clock reference time. The system control center performs time alignment on the data collected by different sub-units based on the spatiotemporal stamp information, associating kinematic data and physiological state data collected at the same time. The data fusion preprocessing unit receives the time-aligned raw data, performs data cleaning, data filtering, and data standardization, and generates a multidimensional feature vector sequence containing spatiotemporal stamps, which is then stored in the data buffer.

[0045] Step 2: Physiological signal feature extraction and analysis. Feature extraction and state analysis are performed on the physiological state data. Independent component analysis algorithm is used to separate the feature components related to muscle tension from the electromyography signal. Adaptive filtering algorithm is used to extract respiratory frequency and respiratory depth parameters from the respiratory airflow pressure signal. Wavelet transform algorithm is used to perform multi-scale decomposition of the center of gravity displacement signal to obtain the center of gravity stability index. Multi-signal fusion framework is used to fuse the extracted feature components, frequency parameters and stability index at the feature level to generate a physiological feature vector reflecting the actor's internal physiological state.

[0046] The physiological signal analysis module first preprocesses the raw electromyographic (EMG) signals received by the EMG signal processing unit. The EMG signal processing unit then applies a bandpass filter to the raw EMG signals, with the passband range set from 20 Hz to 500 Hz. A low-frequency cutoff frequency of 20 Hz is set to remove low-frequency interference caused by baseline drift and respiratory movements, while a high-frequency cutoff frequency of 500 Hz is set to remove environmental electromagnetic interference and power frequency noise.

[0047] After bandpass filtering, the electromyographic (EMG) signals are processed by Independent Component Analysis (ICA). ICA decomposes the observed multi-channel EMG signals into several statistically independent source signals, and then separates the feature components related to muscle tension from these source signals. The specific processing steps include constructing an observation matrix from the multi-channel EMG signals, centering the observation matrix, calculating the covariance matrix of the observation matrix, performing eigenvalue decomposition on the covariance matrix to obtain eigenvectors and eigenvalues, selecting the eigenvectors corresponding to the top principal components according to the eigenvalues ​​to construct a projection matrix, projecting the observation matrix onto the projection matrix to obtain the independent components, and identifying and extracting the feature components related to muscle tension from these independent components.

[0048] The electromyography (EMG) signal processing unit uses time-domain statistical analysis to calculate the quantitative index of muscle tension. The specific processing procedure is as follows: First, the preprocessed EMG signal is segmented, dividing the continuous EMG signal sequence into fixed-length time windows with 50% overlap between adjacent time windows to ensure the continuity of feature extraction. For the EMG signal sequence within each time window, its root mean square (RMS) value is calculated: first, the square of each sampling point value in the signal sequence is obtained, then the squared sequences are summed within the time window to obtain the total power, and finally, the square root of the total power is taken to obtain the RMS value. The magnitude of the RMS value is positively correlated with the muscle contraction intensity; the stronger the muscle contraction, the larger the RMS value of the EMG signal. Simultaneously, the integral EMG value is calculated: first, the absolute value of each sampling point value in the signal sequence is calculated, then the absolute value sequence is numerically integrated within the time window to obtain the integral value. The magnitude of the integral EMG value is positively correlated with the cumulative degree of muscle activity; the longer the duration of muscle activity, the larger the integral EMG value.

[0049] After receiving the respiratory airflow pressure signal output by the respiratory airflow sensing unit, the respiratory signal processing unit first preprocesses the original respiratory signal. The respiratory signal processing unit applies low-pass filtering to the original respiratory signal, and the cutoff frequency of the low-pass filter is set to 5 Hz to extract the low-frequency components related to breathing and remove high-frequency noise interference.

[0050] The low-pass filtered respiratory signal is then processed by an adaptive filtering algorithm. This algorithm automatically adjusts the filter parameters based on the statistical characteristics of the respiratory signal to optimally extract respiratory rate and depth parameters. The adaptive filtering algorithm's process includes initializing the filter coefficients, setting the filter order and convergence factor, comparing the filter output with the desired signal to minimize the mean square error, and iterating the above process until the filter converges and optimally extracts the respiratory rate and depth information from the respiratory signal.

[0051] The respiratory signal processing unit calculates respiratory cycle parameters using a peak detection algorithm. The specific processing steps are as follows: First, the low-pass filtered respiratory signal is smoothed to eliminate false peaks caused by noise. Smoothing is achieved using a moving average filter, with the window length set to one-twentieth of the respiratory signal sampling rate. For the smoothed respiratory signal, all local maxima are identified as candidate respiratory peaks. A local maxima is determined when its value is greater than the values ​​of its immediate neighbors. All candidate peaks are sorted by value, and the top few with the largest values ​​are selected as valid respiratory peaks. The time interval between adjacent valid respiratory peaks is calculated as the respiratory cycle. The time segment corresponding to the complete process of the signal rising from a trough to a peak and then falling back to a trough between adjacent respiratory peaks is calculated. This time segment is divided into an inspiratory phase and an expiratory phase. The inspiratory phase is the time from the trough to the peak, and the expiratory phase is the time from the peak to the trough. The time interval between the troughs between adjacent respiratory peaks is calculated as the respiratory interval parameter.

[0052] After receiving the center of gravity displacement signal output by the inertial measurement unit, the center of gravity signal processing unit performs multi-scale decomposition on the signal using a wavelet transform algorithm. In this embodiment, the db4 wavelet is selected as the wavelet basis function, and the number of decomposition layers is set to 5. The center of gravity displacement signal is decomposed into one approximate component and five detail components. The center of gravity stability index is obtained by analyzing the energy distribution of the detail components at each scale. The specific processing steps include selecting the wavelet basis function and the number of decomposition layers, performing convolution operations between the center of gravity displacement signal and the wavelet basis function to obtain wavelet coefficients, dividing the wavelet coefficients into different scale layers according to frequency from low to high, and analyzing the energy distribution of the wavelet coefficients at each scale layer. The higher the energy of the high-frequency layer, the more violent the center of gravity fluctuation; the higher the proportion of energy in the low-frequency layer, the more stable the center of gravity.

[0053] The center-of-gravity signal processing unit evaluates the stability of the center of gravity by calculating the standard deviation. The specific processing procedure is as follows: First, the component signals of the center-of-gravity displacement signal in the three coordinate axes are extracted, and the standard deviation of each component signal within a sliding time window is calculated. The length of the sliding time window is set to one-tenth of the center-of-gravity signal sampling rate to ensure statistical reliability. The standard deviation is calculated as follows: first, the arithmetic mean of all sampling points within the sliding time window is calculated; then, the squared deviation of each sampling point from the arithmetic mean is calculated; the sum of all squared deviations is divided by the total number of sampling points to obtain the variance; finally, the square root of the variance is taken to obtain the standard deviation. The standard deviations in the three coordinate axes are weighted and summed to obtain a comprehensive center-of-gravity stability index. The weighting coefficients are set according to the contribution of each coordinate axis to the stability of the center of gravity. A larger standard deviation indicates a greater center-of-gravity offset and poorer center-of-gravity stability.

[0054] The multi-signal fusion framework integrates the feature information output from three signal processing units, fusing muscle tension quantification indicators, respiratory cycle parameters, and center of gravity stability indicators at the feature level. The framework first normalizes the features of each channel to eliminate dimensional differences. The normalized features are then combined according to preset fusion rules to generate a physiological feature vector reflecting the actor's internal physiological state. Each dimension of the physiological feature vector corresponds to a physiological feature parameter, accurately representing the intensity of the actor's physiological driving force during action execution, providing an objective quantitative basis at the physiological level for determining emotional state.

[0055] Step 3: Dual-stream fusion for emotion state determination. This step involves deeply fusing physiological feature vectors with action feature parameters extracted from kinematic data to construct an emotion state determination model. A dual-stream spatiotemporal graph convolutional network architecture is used to process the spatial topological relationships and temporal series changes of skeletal joints to extract action type features, and to process the nonlinear mapping of physiological feature vectors to extract physiological driving features. In the feature fusion layer, the contribution ratio of the two features is dynamically weighted through an attention mechanism to output the emotion state classification result.

[0056] The emotion state determination module first receives the 3D coordinate sequence of skeletal joints output by the kinematic data processing module from the first stream processor in the dual-stream spatiotemporal graph convolutional network processor. The first stream processor constructs a skeletal spatiotemporal graph structure, using skeletal joints as nodes in the graph and skeletal connections as edges, defining the adjacency matrix of the nodes based on the anatomical connections of the actor's skeleton.

[0057] The specific construction process of the skeletal spatiotemporal graph is as follows: First, define a set of skeletal joints. Each element in the set corresponds to a joint in the human skeletal model. The joint types include 17 joints: head node, neck node, left and right shoulder nodes, left and right elbow nodes, left and right wrist nodes, spine node, left and right hip nodes, left and right knee nodes, and left and right ankle nodes.

[0058] Then, define a set of skeletal connections, where each element is a pair of joints with anatomical connections. These connections include the head node connecting to the neck node, the neck node connecting to the spine node, the spine node connecting to the left and right shoulder nodes, the spine node connecting to the left and right hip nodes, the left and right shoulder nodes connecting to the left and right elbow nodes, the left and right elbow nodes connecting to the left and right wrist nodes, the left and right hip nodes connecting to the left and right knee nodes, and the left and right knee nodes connecting to the left and right ankle nodes.

[0059] Construct an adjacency matrix based on the set of joints and the set of connections. The adjacency matrix is ​​a symmetric matrix, and the matrix elements indicate whether there is a connection between corresponding pairs of joints. If there is a connection, the matrix element is 1; otherwise, it is 0.

[0060] The first stream processor employs multi-scale spatial convolutional kernels to extract spatial relationship features between key nodes. These kernels contain multiple kernels with different receptive field sizes, capturing spatial relationship features between local neighboring nodes, intermediate-range nodes, and global nodes, respectively. The specific configuration of the multi-scale spatial convolutional kernels is as follows: It includes three kernels with different receptive field sizes. The first kernel's receptive field is set to directly adjacent key nodes, capturing local spatial relationship features between adjacent key nodes. The second kernel's receptive field is set to key node pairs separated by one key node, capturing spatial relationship features between intermediate-range key nodes. The third kernel's receptive field is set to all key nodes, capturing global spatial relationship features. The outputs of the three kernels are concatenated along the channel dimension to obtain the final output of the multi-scale spatial convolution.

[0061] The first stream processor employs a temporal convolution kernel to extract the temporal evolution features of joint motion. The temporal convolution kernel performs a one-dimensional convolution operation on the skeletal joint sequence along the time dimension, capturing the motion changes of the same joint at different time steps and the motion correlations between different joints at adjacent time steps. The specific configuration of the temporal convolution kernel is as follows: a one-dimensional convolution operation is used, with a kernel width of 9, a stride of 1, and zero padding at both ends of the time dimension to ensure that the length of the output time series is consistent with the length of the input time series. The temporal convolution kernel slides along the time dimension to perform convolution operations on the joint motion sequence, with the number of channels of the kernel set to be consistent with the input feature dimension to achieve feature transformation.

[0062] By alternating and stacking multi-scale spatial convolution and temporal convolution, the first stream processor ultimately extracts high-level semantic feature vectors that represent action types.

[0063] After receiving the physiological feature vector output by the physiological signal parsing module, the second stream processor performs nonlinear mapping through a multilayer perceptron. The multilayer perceptron consists of an input layer, at least one hidden layer, and an output layer, with information transmitted between layers via fully connected connections. The specific configuration of the multilayer perceptron is as follows: it contains three fully connected layers. The first fully connected layer has its input dimension set to the dimension of the physiological feature vector and its output dimension set to 128. This layer performs a preliminary nonlinear transformation on the input features. The second fully connected layer has its input dimension set to 128 and its output dimension set to 64. This layer further abstracts and compresses the features. The third fully connected layer has its input dimension set to 64 and its output dimension set to 32. This layer maps the features to the physiological driving feature space to obtain the physiological driving feature vector.

[0064] The input layer receives all dimensions of the physiological feature vector as input features. The hidden layer performs a nonlinear transformation on the input features. Each neuron in the hidden layer applies an activation function after weighted summation of the outputs of all neurons in the previous layer. The activation function uses a modified linear unit function to enhance the nonlinear expressive power of the network. The output layer maps the features output by the hidden layer to the physiological driving feature space, obtaining a physiological driving feature vector representing the physiological driving state.

[0065] The feature fusion layer receives the action type features output by the first stream processor and the physiological driving features output by the second stream processor, and then uses a soft attention mechanism to calculate the attention weights of the two features. The specific implementation process of the soft attention mechanism is as follows: First, the action type features from the first stream processor and the physiological driving features from the second stream processor are input into two independent fully connected layers, each with an output dimension of 1. The output values ​​of these layers are activated by a sigmoid function to obtain attention scores. The two attention scores are then normalized using a softmax function to obtain attention weights, which satisfy the condition that the sum of their weight values ​​is 1. The feature fusion layer uses a weighted summation method, multiplying the action type features by the first attention weight and then adding them together with the physiological driving features multiplied by the second attention weight to obtain the fused emotional feature representation.

[0066] The numerical range of attention weights is between 0 and 1, and the sum of all attention weights is 1. The larger the value of the attention weight, the greater the contribution of that type of feature to the final classification result.

[0067] The fused emotional features are input into an emotional state classifier for emotional state classification. The classifier outputs classification probabilities for four distinct emotional states: calm gaze, passionate exuberance, somber farewell, and hesitant hesitation. The system selects the emotional state with the highest classification probability as the final result and outputs an emotional state label.

[0068] As a further implementation method, the training configuration of the dual-stream spatiotemporal graph convolutional network is as follows: the training adopts the cross-entropy loss function, the sentiment labels labeled by experts are used as the supervision ground truth, the optimizer adopts the Adam algorithm, the initial learning rate is set to 0.001, and the batch size is set to 32; the training is considered to have converged when the classification accuracy of the validation set improves by less than 0.5% for 20 consecutive rounds.

[0069] Step 4: Dynamic generation of emotion-driven special effects parameters. Based on the emotion state classification results, the particle special effects generation parameters are dynamically selected and adjusted. A mapping rule library from emotion state to particle special effects parameters is constructed. Each emotion state corresponds to a set of special effects generation parameters, including particle shape parameters, light effect color parameters, motion trajectory parameters, and life cycle parameters.

[0070] The special effects parameter generation module first looks up the corresponding special effects generation parameter set from the mapping rule base storage based on the emotion state classification results. The mapping rule base stores the correspondence between four emotion states and special effects parameter sets, with each emotion state corresponding to a complete set of special effects generation parameters.

[0071] For the calm gaze emotional state, the corresponding special effect generation parameters are configured as follows: the particle shape parameter is configured as a soft halo shape, the particle appearance shape is set to a circular light spot, the edge of the light spot is set to a gradient blur effect to form a soft halo effect, and the particle number range is set to 100 to 300 particles.

[0072] The lighting effect color parameters are configured as a warm gradient spectrum, with the main color of the particles set to a gradient from warm yellow to light orange. The gradient curve is set to a smooth transition to create a gentle and comfortable visual experience. The motion trajectory parameters are configured as a slow upward floating trajectory, with the initial position of the particles set around the actor's hands. The direction of particle movement is set to a slow upward drift, and the movement speed is set to 5 to 10 centimeters per second. The lifespan parameters are configured as a long decay duration, with the particle survival time set to 3 to 5 seconds. The decay method is set to linear decay to maintain the persistence of the particle effect.

[0073] For the passionate and unrestrained emotional state, the corresponding special effects generation parameters are configured as follows: Particle shape parameters are configured as bright and flashing, the particle appearance shape is set to star-shaped light spots, the light spots are set to a periodic flashing effect, and the particle quantity range is set to 300 to 500 particles. Light effect color parameters are configured as a vibrant gradient spectrum, the main color of the particles is set to a rapid gradient from red to yellow, and the gradient curve is set to rapid jumps to express a sense of vitality.

[0074] The motion trajectory parameters are configured as a rapid diffusion trajectory, with the particle's direction of motion set to rapidly diffuse outwards from the center, and the motion speed set to 20 to 40 centimeters per second. The lifespan parameters are configured as a short decay time, with the particle's survival time set to 1 to 2 seconds, and the decay method set to exponential decay to represent an instantaneous burst.

[0075] For the heavy farewell emotional state, the corresponding special effect generation parameters are configured as follows: the particle shape parameter is configured as a dim and dissipated form, the particle appearance shape is set to a light gray circular light spot, the light spot transparency is set to a low level to express the dimness, and the particle number range is set to 50 to 150 particles.

[0076] The light effect color parameters are configured as a cool-toned decay spectrum, with the particle's main color tone set to a gradient from dark blue to grayish-black, and the gradient curve set to gradually darken to represent a sense of fading away. The motion trajectory parameters are configured as a slow sinking and dissipation trajectory, with the particle's motion direction set to a slow downward descent, and the motion speed set to 2 to 5 centimeters per second. The lifespan parameters are configured as a rapid decay duration, with the particle's survival time set to 0.5 to 1 second, and the decay method set to rapid decay to represent a sense of annihilation.

[0077] For the emotional state of hesitation and indecision, the corresponding special effects generation parameters are configured as follows: Particle shape parameters are configured as an unstable flickering pattern, with the particle appearance shape set to irregular polygonal light spots, the light spots set to an unstable flickering effect, and the particle quantity range set to 100 to 200 particles. Light effect color parameters are configured as a neutral transition spectrum, with the particle's main color tone set to a gradient from white to light gray, and the gradient curve set to slow alternation to express hesitation. Motion trajectory parameters are configured as an irregular oscillating trajectory, with the particle's movement direction set to irregular left and right oscillations, and the movement speed set to 5 to 15 centimeters per second. Lifespan parameters are configured as a medium decay duration, with the particle's survival time set to 2 to 3 seconds, and the decay method set to pulsating decay to express instability.

[0078] The special effects parameter generation module retrieves the corresponding set of special effects generation parameters by looking up the table based on the received emotional state tags. Then, it parses the particle shape parameters, light effect color parameters, motion trajectory parameters, and life cycle parameters and converts them into a parameter format that the particle engine can recognize. Finally, it outputs the parsed special effects generation parameters to the real-time rendering engine module.

[0079] As a further implementation method, the offline training process for establishing the mapping rule base is as follows: The training data contains a large number of actor movement samples and their corresponding expert-annotated emotional labels and special effects parameter configurations. The training data is obtained by recording the performance movements of multiple actors in different emotional states in a digital performing arts system. For each performance movement, kinematic data and physiological state data are simultaneously collected by a multimodal perception hardware module. After viewing the performance, an expert group labels the corresponding emotional state category and provides suggestions for special effects parameter configurations.

[0080] The training process employs reinforcement learning algorithms to optimize special effects parameters to maximize audience emotional resonance scores. The reinforcement learning uses a deep Q-network framework. The state space is defined as the combination of the current emotional state label and special effects parameters, while the action space is defined as the adjustment increment of the special effects parameters. The reward function is defined as the weighted sum of the audience emotional resonance score and the special effects rendering delay, with the resonance weight set to 0.8 and the delay penalty weight set to 0.2. Training uses a 10% increase in the average audience score on the validation set as the convergence threshold. The mapping rule base ultimately stores four emotional states and their corresponding sets of special effects parameters. The set of special effects parameters for each emotional state includes the particle number range, particle size range, color gradient curve, motion speed range, and lifespan range.

[0081] Step 5: Real-time particle effects rendering and output. Based on the effect generation parameters, the particle engine is driven to perform real-time effects rendering. The physics-based particle system simulates the generation, movement, collision and destruction of particles. A deferred shading architecture is used for real-time lighting effect calculation. The rendering output is pushed in real time through the content distribution network of the digital performing arts platform.

[0082] The real-time rendering engine module first receives the special effects generation parameters output by the special effects parameter generation module. The particle engine initializes the particle system state according to the special effects generation parameters, including setting the position and direction of the particle emitter, setting the initial number range of particles, setting the shape and color parameters of particles, setting the movement speed and direction parameters of particles, and setting the life cycle and decay parameters of particles.

[0083] The particle engine employs a computation shader-based parallel particle simulation architecture to execute the generation, movement, and destruction of particles. A computation shader is a general-purpose computational program that runs on a graphics processing unit (GPU) and can utilize the GPU's massively parallel computing capabilities to simultaneously handle the state update calculations of a large number of particles.

[0084] The specific implementation of the parallel architecture for computational shaders is as follows: A unified computational architecture is adopted, containing multiple parallel computational units, each responsible for processing the state update calculations of several particles. Particle state information is stored in the global memory of the graphics processor, including the particle's position coordinates, velocity vector, color value, lifecycle value, and size value. All computational units are synchronized according to synchronization barriers to ensure data consistency. At the beginning of each simulation time step, all computational units simultaneously read the particle state information, calculate the particle state update amount, and simultaneously write the updated state information.

[0085] The parallel particle simulation architecture stores the state information of all particles in the global memory of the graphics processor and launches multiple parallel computing threads to simultaneously calculate the physical state updates of each particle. At each simulation time step, the particle engine first calculates the generation conditions of each particle. When a particle reaches the upper limit of its life cycle, it is marked as dead. When the particle emitter meets the emission conditions, a new particle is generated and its state parameters are initialized.

[0086] The particle engine calculates the particle's position update based on motion trajectory parameters, calculates the effects of gravity, air resistance, and random perturbations on the particle based on the physics model, and detects collisions between the particle and scene boundaries or other particles using a collision detection algorithm and calculates the collision response. After all particle state update calculations are completed, the particle engine passes the particle state information to the renderer for rasterization processing.

[0087] The renderer uses a deferred shading architecture to perform real-time lighting effect calculations. The specific implementation of the deferred shading architecture is as follows: First, a geometry processing stage is performed, rendering all particle geometry in the scene to multiple render targets. Each render target stores a kind of geometric attribute information, including the normal render target storing the normal vector of each pixel, the depth render target storing the depth value of each pixel, and the base color render target storing the color value of each pixel.

[0088] Then, the lighting processing stage is performed. For each screen pixel, its corresponding normal vector, depth value, and base color value are read. Based on the read geometric attribute information, the lighting result for that pixel is calculated, including the calculation of diffuse lighting components, specular lighting components, and ambient lighting components. Finally, the output of the lighting processing stage is combined with post-processing effects, including screen space ambient occlusion effect processing and Bloom halo effect processing.

[0089] The deferred shading architecture postpones lighting calculations to screen space. First, scene geometry is rendered into a geometry buffer, storing the normal vector and depth value of each pixel. Then, lighting calculations are performed on the pixels in the geometry buffer to generate the final color values. The advantage of deferred shading is that the complexity of lighting calculations is proportional to the screen resolution and independent of the scene's geometric complexity, enabling efficient real-time lighting calculations for a large number of particles. The rendering pipeline supports screen-space ambient occlusion to enhance the spatial depth of particle effects and Bloom effects to enhance the brightness and dreamlike quality of particle effects. The renderer outputs rendered frames at a rate of 60 frames per second, with rendering latency controlled within 16 milliseconds, and supports 4K ultra-high-definition resolution.

[0090] The rendered video stream is pushed in real-time to terminal display devices via the content delivery network (CDN) of the digital performing arts platform. This CDN consists of multiple edge server nodes, deployed near user terminals to reduce network latency. Rendered video frames are first uploaded to a central server for content review and quality control, and then distributed to the edge server nodes for real-time streaming by user terminals. Upon receiving the video stream, the user terminal decodes and displays it, allowing viewers to perceive particle effects that precisely match the actors' movements and emotional states with low latency.

[0091] Example 2: This embodiment is basically the same as embodiment 1, except that this embodiment has been adapted for large-scale outdoor music festival scenarios and is suitable for real-time particle effects generation applications in large-scale outdoor music festival scenarios.

[0092] The system deployed in this application scenario has been adjusted based on the system architecture described in Example 1 as follows: the optical motion capture unit in the multimodal perception hardware module has been expanded to 16 infrared motion capture cameras to cover a larger performance area. The spatial positioning accuracy of each camera remains at the sub-millimeter level, but the field of view coverage has been expanded to 120 degrees. The breathing airflow sensing unit has added a gas flow sensor to simultaneously collect the airflow signal exhaled from the mouth to determine the vocal state. The inertial measurement unit has added a head posture sensor to collect the actor's head orientation information.

[0093] Among them, the vocal intensity parameter is obtained after feature extraction of the oral exhalation airflow signal. It is used as an auxiliary feature input to the second stream processor of the emotion state determination module. After being concatenated with the physiological feature vector, it participates in the physiological driving feature extraction. The higher the vocal intensity, the higher the weight of the arousal dimension in the emotion determination.

[0094] The emotional state determination module has expanded the emotional state categories for music festival scenarios. In addition to the four emotional state categories described in Example 1, two new emotional state categories, "Exhilaration" and "Climax," have been added. The special effects generation parameters for the "Exhilaration" emotional state are configured as follows: particle shape parameter set to rapid rotation, light effect color parameter set to a golden gradient spectrum, motion trajectory parameter set to a spiral upward trajectory, and lifespan parameter set to a medium decay duration. The special effects generation parameters for the "Climax" emotional state are configured as follows: particle shape parameter set to an explosive diffusion pattern, light effect color parameter set to a red-white gradient spectrum, motion trajectory parameter set to a high-speed outward diffusion trajectory, and lifespan parameter set to a short decay duration.

[0095] The real-time rendering engine module has been adapted for outdoor display terminals, supporting 8K ultra-high-definition resolution to match the needs of large outdoor screens, and increasing the rendering frame rate to 120 frames per second to ensure smooth high-dynamic-range images. The content delivery network has added adaptive bitrate adjustment to adapt to video transmission needs under different network bandwidth conditions.

[0096] Example 3: This embodiment is basically the same as embodiment 1, except that this embodiment has been adapted for immersive virtual reality performance scenarios and is suitable for immersive particle effects presentation application scenarios for users of virtual reality headsets.

[0097] The system deployed in this application scenario has been adjusted based on the system architecture described in Example 1 as follows: The emotion state determination module adds a virtual perspective motion capture data processing path. This path receives posture data from the virtual reality controller as auxiliary kinematic input, fuses it with skeletal joint motion data collected by the optical motion capture unit, and then inputs it into a dual-stream spatiotemporal graph convolutional network for processing to improve the accuracy of motion feature extraction. Specifically, the virtual perspective motion capture data and the optical motion capture data are fused using a feature layer fusion method. After each data point is processed by its respective feature extraction network to output motion feature vectors, they are weighted and fused with a weight ratio of 0.3:0.7. The fused feature vector is then input into the spatiotemporal graph convolutional network of the first stream processor for further processing.

[0098] The real-time rendering engine module has been adapted for virtual reality display devices. The rendering pipeline uses a forward shading architecture instead of a deferred shading architecture to reduce rendering latency to within 8 milliseconds, meeting the stringent requirements of virtual reality applications for motion-to-image latency. The renderer adds stereoscopic rendering capabilities to simultaneously generate rendering images from both the left and right eye perspectives. The left and right eye images are adjusted for parallax based on the interpupillary distance parameters of the virtual reality headset to create a sense of depth. The particle engine has been optimized for the near-eye display characteristics of virtual reality scenes. The particle rendering size adaptively scales based on the distance between the particles and the virtual camera to avoid particles being too small to be discernible or too large to cause occlusion. Spatial audio synchronization has been added to particle effects; the generation, movement, and disappearance of particle effects trigger corresponding sound events synchronously to enhance the immersive auditory experience.

[0099] Comparative example: To clearly illustrate the technical effects of the present invention, a comparative analysis is conducted below with the embodiments of the present invention. This test set includes 240 performance samples from 12 professional actors, covering four emotional states, with 60 samples in each category. The test environment is consistent with the digital performance scene configuration of Embodiment 1, and the accuracy is judged based on the manual annotation results of three performance experts.

[0100] Comparative Example 1: Comparative Example 1 employs a single-modal motion capture particle effects generation scheme from existing technologies. This scheme only collects the skeletal joint motion trajectory information of the actor through optical motion capture units, without collecting physiological state data such as respiratory airflow pressure signals, electromyographic signals, and center of gravity displacement signals. Emotional state determination is based solely on motion feature parameters extracted from kinematic data, using a single-stream spatiotemporal graph convolutional network architecture for motion type classification, without involving the fusion and determination of physiological features and motion features. Particle effects generation simply selects the corresponding effect template from a preset effect template library based on the motion type classification result for rendering output; different motion types correspond to different fixed effect templates.

[0101] The experiments were conducted in the same digital performance setting, with the same group of actors performing according to the same script. Particle effects were generated using the scheme of Embodiment 1 of this invention and the scheme of Comparative Example 1, respectively. The test results showed that for the action type of raising a hand, the scheme of Embodiment 1 could distinguish between a calm gazing emotional state and a somber farewell emotional state based on the actor's physiological state data, generating soft halo particle effects and dim dissipating particle effects, respectively. The scheme of Comparative Example 1 could only recognize the action type of raising a hand, uniformly generating a preset raising hand effect template, failing to distinguish the emotional semantic differences of the same action type under different physiological states, and the generated particle effects deviated significantly from the actor's actual emotional intention.

[0102] Comparative Example 2: Comparative Example 2 employs an emotion state determination scheme lacking a two-stream spatiotemporal graph convolutional network architecture. This scheme also collects kinematic and physiological state data, but in the emotion state determination stage, it replaces the two-stream spatiotemporal graph convolutional network architecture with a simple feature concatenation method. Specifically, it directly concatenates kinematic and physiological feature parameters into a joint feature vector, which is then input into a multilayer perceptron classifier for emotion state classification. This lacks the independent extraction and attention fusion process of action type features and physiological drive features inherent in a two-stream architecture.

[0103] The experimental tests were also conducted in the same digital performance scenario. The test results showed that the classification accuracy of the scheme in Comparative Example 2 was 72%, which was lower than the 91% classification accuracy of the scheme in Embodiment 1 of this invention. Analysis revealed that the direct feature splicing method could not effectively distinguish the contribution weight of action type features and physiological drive features to emotional states, resulting in unstable classification results for the same action type under different physiological states, leading to misclassification.

[0104] Table 1: Comparison of Embodiments of the Invention with Comparative Examples

[0105] Data analysis from the comparison table above shows that the solution in Embodiment 1 of this invention significantly outperforms the solutions in Comparative Examples 1 and 2 in core performance indicators such as emotional state classification accuracy, emotional state category differentiation ability, and particle effect differentiation. Although Comparative Example 1 is slightly better than the solution in Embodiment 1 of this invention in terms of effect generation delay, it cannot meet the demand for accurate emotional expression in digital performance scenarios due to the lack of physiological state perception capability. The solution in Embodiment 1 of this invention, through the organic combination of multimodal motion data acquisition, dual-stream spatiotemporal graph convolutional network architecture, and differentiated effect parameters for emotional state mapping, achieves accurate semantic matching between particle effects and performance emotions, significantly improving the expressiveness and emotional narrative ability of digital performance scenarios.

[0106] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating digital performance particle effects based on multimodal motion capture, characterized in that, Includes the following steps: Step 1: The kinematic and physiological data of the actors are synchronously acquired through a multimodal sensing array deployed in the performance area. The kinematic data includes the three-dimensional spatial position, motion trajectory and angular velocity parameters of the skeletal joints. The physiological data includes respiratory airflow pressure signals, muscle surface electromyography signals and center of gravity displacement signals. The sampling clocks of all subsystems are time-aligned through a unified clock synchronization protocol. The raw data collected is preprocessed by data fusion to generate a multidimensional feature vector sequence containing spatiotemporal stamps. Step 2: Feature extraction and state analysis of physiological state data. Independent component analysis algorithm is used to separate feature components related to muscle tension from electromyography signal. Adaptive filtering algorithm is used to extract respiratory frequency and respiratory depth parameters from respiratory airflow pressure signal. Wavelet transform algorithm is used to decompose the center of gravity displacement signal into multi-scale values ​​to obtain center of gravity stability index. Multi-signal fusion framework is used to fuse the extracted feature components, frequency parameters and stability index at the feature level to generate physiological feature vector reflecting the actor's internal physiological state. Step 3: Deeply fuse the physiological feature vectors with the action feature parameters extracted from the kinematic data to construct an emotional state determination model. A dual-stream spatiotemporal graph convolutional network architecture is used to process the spatial topological relationship and temporal series changes of skeletal joints to extract action type features, and to process the nonlinear mapping of physiological feature vectors to extract physiological driving features. In the feature fusion layer, the contribution ratio of the two features is dynamically weighted through an attention mechanism to output the emotional state classification result. Step 4: Dynamically select and adjust particle effect generation parameters based on the emotional state classification results, and construct a mapping rule library from emotional state to particle effect parameters. Each emotional state corresponds to a set of effect generation parameters, including particle shape parameters, light effect color parameters, motion trajectory parameters, and life cycle parameters. Step 5: Drive the particle engine to perform real-time special effects rendering based on the special effects generation parameters. The physical particle system simulates the generation, movement, collision and destruction of particles. The deferred shading architecture is used to calculate the lighting effects in real time. The rendering output is pushed in real time through the content delivery network of the digital performing arts platform.

2. The method for generating digital performance particle effects based on multimodal motion capture according to claim 1, characterized in that, In step one, the multimodal sensing array includes an optical motion capture subsystem for acquiring motion trajectory information of skeletal joints, a sensing subsystem for acquiring respiratory airflow pressure signals, an acquisition subsystem for acquiring electromyographic signals of muscle surfaces, and an inertial measurement unit.

3. The method for generating digital performance particle effects based on multimodal motion capture according to claim 1, characterized in that, In step two, the root mean square value and integral electromyographic value of the electromyographic signal are calculated using time-domain statistical analysis as quantitative indicators of muscle tension. The inspiratory duration, expiratory duration and respiratory interval within the respiratory cycle are calculated using peak detection algorithm, and the degree of center of gravity stability is evaluated by standard deviation calculation.

4. The method for generating digital performing arts particle effects based on multimodal motion capture according to claim 1, characterized in that, In step three, the first stream of the dual-stream spatiotemporal graph convolutional network uses multi-scale spatial convolution kernels to extract the spatial relationship features of the joints and uses temporal convolution kernels to extract the temporal evolution features of the joint motion. The second stream performs nonlinear mapping through a multilayer perceptron. The feature fusion layer uses a soft attention mechanism to calculate the attention weights of the first and second stream features, and the fusion method is weighted summation.

5. The method for generating digital performing arts particle effects based on multimodal motion capture according to claim 1, characterized in that, In step four, the mapping rule base from emotional state to particle effect parameters is established through offline training. The training data includes action samples, expert-annotated emotional tags, and effect parameter configurations. The training process uses a reinforcement learning algorithm, with the weighted sum of the audience's emotional resonance score and the effect rendering delay as the reward signal to optimize the effect parameters and improve the audience's emotional resonance.

6. The method for generating digital performing arts particle effects based on multimodal motion capture according to claim 1, characterized in that, In step five, the particle engine uses computational shaders to implement parallel particle simulation, supporting real-time computation of more than 500,000 particles. The rendering pipeline adopts a deferred shading architecture, supporting screen-space ambient occlusion and bloom effects.

7. A digital performance particle effects generation system based on multimodal motion capture, applicable to the digital performance particle effects generation method based on multimodal motion capture as described in any one of claims 1-6, characterized in that, It includes a multimodal perception hardware module, a physiological signal analysis module, an emotional state determination module, a special effects parameter generation module, and a real-time rendering engine module; The multimodal perception hardware module is used to synchronously collect the actor's kinematic data and physiological state data. The kinematic data includes the three-dimensional coordinate sequence of skeletal joints, trajectory information and angular velocity parameters. The physiological state data includes respiratory airflow pressure signals, electromyographic signals and center of gravity displacement signals. Temporal synchronization constraints are applied to the two types of data. The physiological signal parsing module is used to separate muscle tension feature components from electromyography signals based on independent component analysis algorithm, extract respiratory rhythm parameters from respiratory airflow pressure signals based on adaptive filtering algorithm, extract center of gravity stability index from center of gravity displacement signals based on wavelet transform algorithm, and output physiological feature vectors. The emotional state determination module has a built-in dual-stream spatiotemporal graph convolutional network, which is used to extract the action type features and physiological driving features of skeletal joints. The contribution ratio of the two types of features is dynamically weighted through an attention mechanism, and the emotional state classification result corresponding to the emotional semantics of the performance scene is output. The special effects parameter generation module is used to generate corresponding particle special effects generation parameters based on the emotional state classification results. The real-time rendering engine module is used to drive the particle engine to perform real-time rendering output based on the special effects generation parameters.

8. The digital performing arts particle effects generation system based on multimodal motion capture according to claim 7, characterized in that, The multimodal sensing hardware module includes an optical motion capture subunit, a respiratory airflow sensing subunit, an electromyography (EMG) signal acquisition subunit, and an inertial measurement unit. The optical motion capture subunit is used to acquire motion trajectory information of skeletal joints, the respiratory airflow sensing subunit is used to acquire respiratory airflow pressure signals, the EMG signal acquisition subunit is used to acquire EMG signals on the muscle surface, and the inertial measurement unit is used to acquire inertial measurement data.

9. The digital performing arts particle effects generation system based on multimodal motion capture according to claim 7, characterized in that, The physiological signal analysis module calculates the root mean square value and integral electromyographic value of the electromyographic signal using time-domain statistical analysis, calculates respiratory cycle parameters using peak detection algorithm, and evaluates the center of gravity stability index using standard deviation.

10. The digital performing arts particle effects generation system based on multimodal motion capture according to claim 7, characterized in that, The first stream of the dual-stream spatiotemporal graph convolutional network uses multi-scale spatial convolution kernels to extract spatial relationship features of key points and temporal convolution kernels to extract temporal evolution features. The second stream is nonlinearly mapped through a multilayer perceptron; The feature fusion layer uses a soft attention mechanism to calculate the attention weights of the two stream features.

Images