Interactive stage audio-visual presentation system fusing audience feedback

Abstract

The invention relates to the field of man-machine interaction and multimedia processing, and discloses an interactive stage audio-visual presentation system fusing audience feedback, which comprises a multi-mode sensing device set, a stage effect execution device set and a software module, an intention and potential field construction engine generates a basic potential field according to audience state data collected by a multi-mode sensing module, and meanwhile, the basic potential field is displayed on a stage effect execution device set; identifying the attitude data of the performer as a topology constraint instruction, and constructing a topology constraint function; the topological constraint function and the basic potential field are subjected to mathematical combination, a final potential field fusing information of the two parties is obtained through calculation, a field feature analysis module carries out differential operation on the final potential field, multiple feature fields such as a gradient field, a divergence field and a rotation field are extracted, a generative audio-visual mapping engine maps the feature fields into a control instruction, and the control instruction is sent to a cloud server; visual and audio content generation is driven. According to the invention, deep fusion of the audience group state and the artistic intention of the performer is realized, and complex and organic emerging type audio-visual experience can be created.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and multimedia processing, specifically to an interactive stage audiovisual presentation system that integrates audience feedback. Background Technology

[0002] With the development of sensor technology, real-time computing and digital media art, interactive stage audiovisual presentation systems play an important role in various occasions such as theater, concerts and exhibitions. These systems attempt to capture dynamic information on site, such as the movements of performers or the voices of the audience, to change stage effects such as lighting, video and sound in real time, in order to break the traditional one-way viewing relationship and create different live experiences.

[0003] However, current mainstream interactive technologies still have limitations in achieving deep human-computer and human-to-human interaction. The interaction logic of these systems is based on relatively direct mapping relationships, such as directly mapping the decibel value of sound to the brightness of light, or recognizing a specific action as an instruction to trigger a preset video. Although this mode can achieve real-time interaction, it is difficult to capture and express the collective emotional atmosphere of the audience as a whole. In addition, the performer's role in these systems is simplified to another input source. Their macroscopic and artistic choreographic intentions, such as guiding the "convergence" or "division" of energy on site, are difficult for the system to understand and meaningfully integrate with the collective state of the audience. This results in the final audiovisual effect being fragmented, predictable, and lacking organic unity, complexity, and emergentness.

[0004] The field requires not only a comprehensive and holistic quantitative perception of the collective state of the audience, but also a deep integration mechanism that applies the performer's higher-order artistic intentions as a structural force to the audience's state. Therefore, this invention proposes an interactive stage audiovisual presentation system that integrates audience feedback to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an interactive stage audiovisual presentation system that integrates audience feedback. This solves the problem that existing interactive stage presentation systems typically rely on pre-set scripts or simple, one-to-one mapping relationships, making it difficult to perceive and integrate the collective emotional state of the audience in real time and holistically. They also cannot dynamically and complexly integrate the performer's macro-artistic intentions with this collective state, resulting in the final audiovisual effect lacking a high degree of correlation with the atmosphere of the scene.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: an interactive stage audiovisual presentation system that integrates audience feedback, comprising: the system includes hardware and software components;

[0007] The hardware component includes a set of multimodal sensing devices for data acquisition and a set of stage effect execution devices for effect presentation.

[0008] The software component is configured in the central processing server and serves as the carrier for implementing the core functions of this invention. The software modules include: a multimodal perception module, an intent and potential field construction engine, a field feature analysis module, and a generative audiovisual mapping engine.

[0009] Preferably, the multimodal sensing module is used to receive heterogeneous raw data from the multimodal sensing device set and perform preprocessing. The multimodal sensing module includes an audience state acquisition unit, which receives temperature data from the thermal imaging array, image data from the visible light camera array, and audio data from the microphone array.

[0010] To integrate data from these different sources, the audience state acquisition unit performs spatial and temporal alignment on the data and normalizes it, ultimately outputting three data fields in a unified coordinate system: a temperature scalar field representing the physiological thermal distribution of the audience area, a motion vector field representing the amplitude of the audience's limb movements, and a sound pressure scalar field representing the energy distribution of the on-site acoustic environment. These three data fields together constitute a quantitative description of the audience group's state, namely, audience state data.

[0011] Preferably, the intention and potential field construction engine is the core innovation of this invention. It is responsible for mathematically fusing the audience's group state with the performer's subjective intention. The intention and potential field construction engine includes a basic potential field generator, a performer intention classifier, and a topological constraint processor. The basic potential field generator receives audience state data and, through a preset linear weighted fusion formula, fuses the temperature scalar field, the motion velocity scalar field calculated from the motion vector field, and the sound pressure scalar field into a basic potential field that can characterize the overall energy distribution of the audience. Its calculation method can be expressed by the following formula:

[0012] ;

[0013] in, , , These are pre-defined non-negative weighting coefficients used to adjust the relative contributions of the three modes: heat, motion, and sound. For a temperature scalar field, For the scalar field of motion velocity, It is a scalar field of sound pressure.

[0014] Preferably, the performer intent classifier receives performer posture data collected by motion capture equipment. By matching real-time posture data sequences with templates in a preset posture library, the classifier can identify the performer's specific intent. In one embodiment, the matching process employs a dynamic time warping algorithm, thereby accurately identifying topological constraint commands with clear topological significance, such as "contraction," "expansion," and "cutting."

[0015] After receiving a topology constraint instruction, the topology constraint processor constructs a corresponding topology constraint function based on the type of instruction. .For example:

[0016] When the instruction is "shrink", the position is as specified by the performer. Centered on the center, construct a Gaussian function, the form of which is as follows: Its formula is:

[0017] ;

[0018] The function is at the center point. The maximum value is obtained at the location. And it smoothly decays outward to 1, thus amplifying the weight of the audience energy in the final potential field in the central region;

[0019] Finally, the engine combines the topological constraint functions with the fundamental potential field to calculate the final potential field. This combination method involves point-by-point multiplication:

[0020] ;

[0021] In this way, the performer's intention, as a topological structure, is applied to the original energy field generated by the audience's state, achieving a fusion of the two.

[0022] Preferably, the field feature analysis module is used to perform in-depth analysis of the final potential field, which contains rich information. This module first analyzes the final potential field... With the original motion vector field Perform point-by-point multiplication to construct the energy flow field This field describes the tendency of energy flow in space. Subsequently, differential geometric operations are performed on the final potential field (a scalar field) and the energy flow field (a vector field) to calculate several characteristic fields, including:

[0023] Gradient field of the final potential field This indicates the direction and rate of the fastest energy change:

[0024] ;

[0025] in, This indicates that the final potential field is at the grid point. The value at that point. For points on the boundary, the forward or backward difference method can be used for calculation, and the output of this calculation is the gradient vector field. .

[0026] divergence field of energy flow field This indicates the source or sink of energy, that is, the region where energy is created or destroyed.

[0027] ;

[0028] in, and These are energy flow fields At grid points place and The components, the output of the calculation is a divergence scalar field. .

[0029] curl field of energy flow field This represents the rotation and vortex intensity of energy.

[0030] .

[0031] Preferably, the generative audiovisual mapping engine is responsible for transforming abstract feature fields into specific stage effects. The generative audiovisual mapping engine maps multiple feature fields into control instructions sent to the stage effect execution device set. The generative audiovisual mapping engine includes a visual content generator. The visual content generator maps the values ​​of feature fields such as gradient field, divergence field, and curl field at various points in space into visual parameters through a preset visual mapping function. The visual parameters are encapsulated as part of the control instructions and sent to the video rendering server for real-time control of the particle system's rendering algorithm. The specific mapping relationship can be as follows: the value of the divergence field controls the emission rate of the particle system, the direction and magnitude of the gradient field control the initial velocity of the particles, the value of the final potential field controls the color and size of the particles, and the curl field defines the force field parameters acting on the particle system.

[0032] Preferably, the generative audiovisual mapping engine further includes a spatial audio synthesizer. The spatial audio synthesizer maps the values ​​of the feature field to audio parameters through a preset audio mapping function. The audio parameters are also encapsulated in control instructions and sent to the audio server for real-time control of audio synthesis and spatialization processing. The specific mapping relationship can be as follows: the average value of the potential field controls the overall gain of the spatial soundscape, the modulus of the gradient field controls the filter cutoff frequency of the synthesizer timbre, the size of the curl field controls the wet / dry ratio of the reverberation effect, and the direction of the energy flow field is used to modulate the dynamic change rate of the sound phase.

[0033] Preferably, the multimodal sensing device set may specifically include: a thermal imaging array, a visible light camera array, a microphone array, and a motion capture device for capturing performers' movements; the stage effects execution device set may specifically include: a lighting control device, the aforementioned audio server, and a video rendering server.

[0034] This invention provides an interactive stage audiovisual presentation system that integrates audience feedback. It has the following beneficial effects:

[0035] 1. This invention uses an intention and potential field construction engine to mathematically combine the basic potential field representing the state of the audience group with the topological constraint function representing the performer's artistic intention, generating a final potential field that integrates information from both sides. This breaks through the simple trigger-response mode of traditional interactive systems and achieves a deep integration between the audience's unconscious collective feedback and the performer's subjective macro-control. This allows the stage audiovisual effects to not only respond to the atmosphere in real time, but also reflect a clear artistic choreography intention, thus enhancing the depth of collaborative creation in interactive art.

[0036] 2. This invention performs differential geometric operations on the final potential field through a field feature analysis module to extract multiple deep feature fields such as gradient field, divergence field, and curl field. It then uses a generative audiovisual mapping engine to map these feature fields into complex parameters of the audiovisual generation algorithm. This surpasses the traditional approach of directly mapping single data into a single effect parameter. By unifying multiple intrinsic structural features within the field to drive the generation system, it generates highly correlated audiovisual content, making the stage effects present a dynamic aesthetic.

[0037] 3. This invention utilizes a multimodal sensing module to fuse data from thermal imaging arrays, visible light camera arrays, and microphone arrays, quantifying them into physical fields of three dimensions: temperature, motion, and sound pressure. These are then weighted and fused into a unified fundamental potential field. Compared to sensing methods that rely on a single channel, this multimodal fusion technique can more comprehensively and robustly capture and quantify the overall state of the audience, avoiding misjudgments caused by biases or interference from a single data source, and ensuring the accuracy and stability of the system's perception of the atmosphere. Attached Figure Description

[0038] Figure 1 This is a system structure block diagram of the present invention;

[0039] Figure 2 This is a schematic diagram of the physical deployment of the system according to the present invention;

[0040] Figure 3 This is a flowchart illustrating the dynamic potential field construction process of the present invention.

[0041] Figure 4 This is a data flow diagram of the field feature analysis module of the present invention;

[0042] Figure 5 This is a schematic diagram of the generative mapping process of the present invention;

[0043] Figure 6 This is a flowchart of the system operation method of the present invention. Detailed Implementation

[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] See attached document Figure 1 An interactive stage audiovisual presentation system that integrates audience feedback includes: a central processing server, a multimodal sensing device set, a stage effects execution device set, and a data exchange network.

[0046] A central processing server is a high-performance computer or server cluster, internally equipped with a central processing unit, graphics processing unit memory, non-volatile storage devices, and network interfaces. The central processing server is used to receive and process data from the multimodal sensing device set, execute dynamic potential field construction and feature analysis algorithms, generate control commands, and send the commands to the stage effects execution device set.

[0047] A multimodal sensing device set is used to collect real-time status data of the audience and performers. The multimodal sensing device set includes:

[0048] The thermal imaging array consists of long-wave infrared thermal imaging cameras deployed above the performance venue, overlooking the audience seating area. The thermal imaging array is used to capture the thermal distribution of the audience seating area and output digital thermal imaging image data containing temperature values ​​at each spatial coordinate point. The thermal imaging array is connected to the data exchange network via a gigabit Ethernet interface.

[0049] A visible light camera array consists of visible light cameras deployed to cover all or key areas of the audience. The visible light camera array is used to capture real-time images of the audience and output digital video stream data. The visible light camera array is connected to a data exchange network via a gigabit Ethernet interface.

[0050] A microphone array, consisting of multiple directional microphones, is used to collect sound signals from the scene and output multi-channel digital audio signals. The microphone array is connected to an audio interface device via the Dante network protocol or a multi-channel audio interface, which in turn connects to a central processing server.

[0051] Motion capture equipment, whether an inertial measurement unit (IMU) system worn on key nodes of the core performer's body or an optical motion capture system deployed within the venue, is used to acquire real-time three-dimensional skeletal posture data of the performer. The motion capture equipment connects to a central processing server via a dedicated wireless or wired receiver through a USB or Ethernet interface.

[0052] A set of stage effects execution devices, used to receive control commands from a central processing server and execute specific stage audiovisual effects, including:

[0053] Lighting control equipment refers to lighting controllers, network nodes, or luminaires that support the Art-Net or sACN protocols. The lighting control equipment receives lighting control commands from the central processing server via an Ethernet interface from the data exchange network and drives the on-site luminaires to perform changes in brightness, color, angle, etc.

[0054] An audio server is a dedicated computer that runs digital audio workstation software or a programmable audio synthesis engine. The audio server receives audio parameter control instructions from the central processing server via the Dante network protocol, and synthesizes, processes or plays audio in real time based on these instructions, and then outputs the audio signal to the on-site sound reinforcement system.

[0055] A video rendering server is a computer equipped with a high-performance graphics processor. The video rendering server receives rendering parameters and instructions from the central processing server via Ethernet from the data exchange network, generates or processes visual content in real time, and outputs the rendered video signal to display devices such as LED screens and projectors on site through SDI (Serial Digital Interface) or NDI (Network Device Interface) protocols.

[0056] The data exchange network, consisting of 10 Gigabit Ethernet switches and related cabling systems, forms the physical data transmission layer of the system. It provides a stable, low-latency data exchange channel for the central processing server, network devices housing multimodal sensing equipment, and network devices housing stage effects execution equipment. All devices requiring network communication connect to the data exchange network through standard network interfaces.

[0057] See attached document Figure 1 The software modules of the system described in this invention are all loaded and run in the aforementioned central processing server. These software modules work together to execute the core technical solution of this invention.

[0058] The software modules include: a multimodal perception module, an intent and potential field construction engine, a field feature analysis module, and a generative audiovisual mapping engine.

[0059] The multimodal sensing module receives and preprocesses the raw data stream from the multimodal sensing device set, transforming the raw data stream into a data structure usable by subsequent modules. The multimodal sensing module includes:

[0060] The audience status acquisition unit receives raw data from the thermal imaging array, visible light camera array, and microphone array, and performs preprocessing operations, including: noise reduction and background temperature correction of the thermal imaging image data; calculation of a two-dimensional motion vector field using an optical flow algorithm on the visible light video stream data; and calculation of the sound pressure level in each area of ​​the audience seating area using a beamforming algorithm on the multi-channel audio signal. The output of the audience status acquisition unit is three spatiotemporally aligned and normalized data fields: a temperature scalar field. Motion vector field , and sound pressure scalar field .

[0061] Performer intent capture unit, used to receive raw pose data from motion capture equipment. It performs coordinate system transformation, converting the data in the local coordinate system of the sensor into the global coordinate system of the stage, and constructs a real-time three-dimensional skeletal model of the performer through inverse kinematics algorithm, outputting a time-series data structure containing the three-dimensional coordinates, rotation and velocity information of each skeletal joint.

[0062] An intention and potential field construction engine is used to generate the final potential field based on the audience's state and the performer's intention, including:

[0063] The performer intent classifier receives skeletal pose time-series data output by the performer intent capture unit. It has a built-in pose library containing multiple topological constraint instruction templates. By matching the real-time input pose sequence with the templates in the library, the performer intent classifier identifies continuous movements as discrete topological constraint instructions. Discrete topological constraint instructions It is a data structure that contains instruction types (such as shrink, cut) and their related parameters.

[0064] The fundamental potential field generator is used to receive the temperature scalar field output by the spectator state acquisition unit. Motion vector field Harmony pressure scalar field According to a preset weighted fusion formula, the three heterogeneous data fields are fused into a single scalar field, namely the fundamental potential field. Basic potential field It represents the distribution of collective energy generated solely by the audience.

[0065] Topology constraint processor, used to receive the fundamental potential field output by the fundamental potential field generator. and topological constraint instructions output by the performer intent classifier And first according to the instructions The content dynamically constructs topological constraint functions that act on the entire field. Subsequently, the topology constraint functions are... With the fundamental potential field By performing point-by-point multiplication, a nonlinear correction is applied to the fundamental potential field to obtain the final potential field. .

[0066] The field feature analysis module is used to analyze the final potential field output by the intent and potential field construction engine. Performs real-time differential geometry operations and extracts its structural and dynamic features, including: a gradient calculation unit for the final potential field. By applying the finite difference method, the gradient vector at each spatial coordinate point is calculated, and the gradient vector field is output. The gradient vector field indicates the direction and rate of the fastest increase in potential energy; the divergence calculation unit is used to calculate the final potential field. With the original motion vector field Multiplication creates an energy flow field. Then, the energy flow field Calculate the divergence and output the divergence scalar field. Characterizes the energy source (positive value region) and sink (negative value region); curl calculation unit, used to calculate the energy flow field. Curl, output curl scalar field , which characterizes the vortex intensity of energy flow.

[0067] The generative audiovisual mapping engine receives the feature fields output by the field feature analysis module and, based on a configurable mapping rule set, converts each feature field into specific control instructions for the stage effect execution device set. Specifically, this includes a visual content generator, which generates the final potential field. gradient field divergence field Spool field These features, through preset mapping functions, are converted into parameters for controlling real-time rendering algorithms such as particle systems, fluid simulations, or procedural textures. These parameters are encapsulated into control messages and sent to the video rendering server via a data exchange network. The spatial audio synthesizer, on the other hand, converts these field features into parameters for controlling audio synthesis and processing through another set of preset mapping functions. These parameters (such as channel gain, filter cutoff frequency, reverberation parameters, and stereo phase modulation depth) are encapsulated into OSC (Open Sound Control) or MIDI (Musical Instrument Digital Interface) messages and sent to the audio server via a data exchange network.

[0068] See attached document Figure 2The multimodal perception module performs the acquisition and preprocessing of this invention, which transforms heterogeneous raw data from different sensors with different physical units, sampling rates and coordinate systems into a unified data structure that has undergone spatiotemporal alignment and normalization, for use by the subsequent intent and potential field construction engine.

[0069] Spatiotemporal alignment and normalization of audience status data: The audience status acquisition unit receives raw data streams from the thermal imaging array, visible light camera array, and microphone array, and performs the following steps:

[0070] Spatial alignment: To ensure that data from different sensors can be mapped to the same physical location in the audience area, the system first performs an offline spatial calibration process, which defines a two-dimensional orthogonal reference coordinate system covering the entire audience area.

[0071] For thermal imaging arrays and visible light camera arrays, the system calculates perspective transformation matrices to map the pixel coordinates of their respective images to a two-dimensional orthogonal reference coordinate system. The transformation matrix is ​​obtained by placing multiple calibration points in the audience area, obtaining their pixel coordinates in the images of each camera and their physical coordinates in the reference coordinate system, and then solving the overdetermined equations using the least squares method.

[0072] The sound pressure distribution map output by the microphone array has been calculated based on the known physical positions of each microphone in the reference coordinate system, so its coordinate system is naturally consistent with the reference coordinate system.

[0073] After spatial alignment, any reference coordinate point Each of these can be mapped to a unique location in the data field of each sensor.

[0074] Time alignment: All data-generating devices in the system, including the central processing server and various sensing devices, are synchronized with a common time server through the network time protocol. All acquired data frames, including thermal imaging images, visible light video frames and audio data blocks, are appended with high-precision timestamps. The audience status acquisition unit has a data buffer to collect data frames with timestamps and to combine the closest data frames into a synchronization data group according to the timestamps for subsequent processing.

[0075] Data Processing and Normalization: For each synchronized data set, the system performs Gaussian filtering on the thermal imaging image data to reduce noise and subtracts the pre-determined ambient background temperature. For the visible light video frames, a dense optical flow algorithm (such as the Farnebäck algorithm) is executed to calculate the motion vector of each pixel, forming a motion vector field. Beamforming algorithms (such as delay-sum beamforming) are performed on multi-channel audio signals to calculate the sound pressure level at each point on the reference coordinate system grid, forming a sound pressure scalar field. .

[0076] Subsequently, to eliminate dimensional differences, the system normalizes each data field, mapping its numerical range to an interval. The normalized temperature scalar field The calculation method is as follows:

[0077] ;

[0078] in, This is the corrected temperature value. and These are the preset lower and upper limits of temperature normalization. The function will value Limited to the range Inside.

[0079] Normalized scalar field of motion velocity The calculation method is as follows:

[0080] ;in, It is the magnitude of the motion vector. It is the preset maximum motion rate.

[0081] Normalized sound pressure scalar field The calculation method is as follows:

[0082] ;in, It is the calculated sound pressure level. and These are the preset lower and upper limits of sound pressure normalization.

[0083] After the above steps, the audience state acquisition unit finally outputs a temperature scalar field, a motion vector field, and a sound pressure scalar field that have been spatiotemporally aligned and normalized.

[0084] The performer intent capture unit receives raw pose data from the motion capture device. And perform the following steps:

[0085] Coordinate system transformation: The original skeletal joint coordinates output by the motion capture device are located in its own local coordinate system. The system determines the transformation relationship between the local coordinate system and the stage's global reference coordinate system through offline calibration. The transformation relationship is determined by a 4x4 affine transformation matrix. This indicates that for any original joint point's three-dimensional coordinate vector Its coordinates in the global reference coordinate system The following matrix multiplication method was used to calculate: ;

[0086] in, and They are and The homogeneous coordinates are used to represent the transformation, which is applied to all joints of the performer's skeletal model.

[0087] Key pose feature extraction provides more representative input to the subsequent intent classifier. The system not only uses the transformed joint coordinates but also extracts high-dimensional key pose features from them, including:

[0088] Limb end effector status: Calculate the three-dimensional position, velocity, and acceleration of the performer's hands, feet, and head in the global reference coordinate system.

[0089] Major joint angles: Calculate the bending angles of major joints such as the elbow, shoulder, knee, and hip.

[0090] Body center of mass: Calculate the position of the center of mass of the entire body model in the global reference coordinate system based on the mass ratio of each body part.

[0091] Posture symmetry: Calculate the rate of change of distance between corresponding joints on the left and right sides.

[0092] Body bounding box volume: Calculates the volume of the smallest axis-aligned bounding box that can completely enclose the performer's 3D skeletal model. The volume of the smallest axis-aligned bounding box reflects the body's extension and contraction state.

[0093] These extracted features together form a high-dimensional feature vector, which, along with the timestamp, serves as the final output of the performer intent capture unit and is passed to the performer intent classifier.

[0094] See attached document Figure 3 The intention is to execute the dynamic potential field construction of this invention with the potential field construction engine, receiving preprocessed audience state data and performer posture data, and constructing a final potential field accordingly. The construction process includes: the fusion generation of the basic potential field, the classification of the performer's intention, the mathematical construction of the topological constraint function, and the calculation of the final potential field.

[0095] The fundamental potential field generator receives the normalized temperature scalar field output from the spectator state acquisition unit. Scalar field of motion speed and sound pressure scalar field Furthermore, by employing a linear weighted fusion formula, the three scalar fields mentioned above are fused into a fundamental potential field representing the original collective energy distribution of the audience. The calculation formula is as follows:

[0096] ;

[0097] in, , , These are pre-defined non-negative weighting coefficients used to adjust the relative contributions of the three modes: heat, motion, and sound.

[0098] The performer intent classifier receives a time series of high-dimensional pose feature vectors output by the performer intent capture unit. The performer intent classifier identifies the performer's intent by matching the real-time input pose sequence with a pre-recorded and stored pose library containing multiple intent templates.

[0099] In one specific embodiment, the matching process employs the Dynamic Time Warping (DTW) algorithm. First, in the offline phase, one or more standard actions are recorded for each intent (e.g., contraction, expansion), and the corresponding feature vector sequence is stored as a template in the pose library.

[0100] During real-time operation, the system maintains a list containing the most recent... A sliding window of frame pose feature vectors, used as a query sequence For any template sequence in the pose library The system calculates the DTW distance between the two, which is achieved by constructing... Cumulative cost matrix To calculate the matrix elements The calculation formula is:

[0101] ;

[0102] in, Two multidimensional feature vectors and The Euclidean distance between them, the final DTW distance is .

[0103] When query sequence With a certain template When the DTW distance between the performers is less than a preset threshold, the system determines that the performer has executed the intent corresponding to the template. Subsequently, the system extracts the parameters necessary to execute the intent from the pose sequence and encapsulates the identified instruction type and the extracted parameters together into a topological constraint instruction. The output is sent to the topology constraint processor.

[0104] The topology constraint processor processes the received topology constraint instructions. To dynamically generate a topological constraint function for the current moment. ,function It is a scalar field acting on the entire audience seating reference coordinate system, used to adjust the fundamental potential field. The weight of energy at each point in the system.

[0105] For different types of instructions Topological constraint functions The specific mathematical form is as follows:

[0106] For the Constrict command: includes the center of action. ,strength and radius of influence , Constructed as The Gaussian function centered at x is given by the following formula:

[0107] ;

[0108] The function is at the center point. The maximum value is obtained at the location. It then smoothly decays outwards to 1, thus amplifying the weight of the audience energy in the final potential field.

[0109] For the Sever instruction: it includes points and Defined line segment and cutting width , Constructed as a line segment A smooth energy suppression band forms nearby, and its formula is:

[0110] ;

[0111] in, It is a point to line segment The minimum Euclidean distance, It is related to the cutting width The relevant smoothness parameters, It is a small positive number close to 0 (e.g., 0.01) used to define the energy residue coefficient within the suppression band. The function value is on the line segment... Up close And smoothly transition to 1 on both sides.

[0112] In some embodiments, performer intentions (such as agitation) are not achieved by modifying the scalar potential field, but rather by generating a vector field to directly influence the energy flow in subsequent calculations. In this case, the topological constraint function... At this moment, the value is always equal to 1, and the system will generate a vector adjustment field. And pass it to the field feature analysis module 0.

[0113] Topology constraint processors use the fundamental potential field Topological constraint functions generated at the current time By performing point-by-point multiplication, the final potential field can be calculated. The calculation formula is:

[0114] ;

[0115] This calculation process is executed in real time during each processing cycle of the system (corresponding to each synchronous data group), thus, the final potential field... It can instantaneously reflect the combination of the audience's collective state and the performer's guiding intention. When no performer's intention is recognized, the topological constraint function... It is always equal to 1, at this time equal .

[0116] The field feature analysis module performs field feature analysis and receives the final potential field output by the intent and potential field construction engine. Differential geometric operations are performed to extract multiple feature fields used to drive generative art.

[0117] See attached document Figure 4 The field feature analysis module performs the field feature analysis of this invention and receives the final potential field output by the intent and potential field construction engine. Differential geometric operations are then performed on it to extract multiple feature fields used to drive generative art, which are executed on a discrete two-dimensional grid that corresponds to the reference coordinate system of the audience seating.

[0118] To calculate the divergence and curl characteristics that reflect the direction and trend of energy flow, the system first constructs the energy flow field. The energy flow field will affect the final potential field. Compared with the raw motion vector field provided by the audience state acquisition unit Defined by point-by-point multiplication, this calculation is performed within the divergence and curl calculation units, and its formula is as follows:

[0119] ;

[0120] Where "·" represents the multiplication of a scalar and a vector, the physical meaning of the calculation is: in regions where the audience's energy is higher (i.e., The larger the value in the area), the physical motion of the audience (by...) (This indicates) the greater the contribution to the overall flow of energy, It is a two-dimensional vector field, and its components are denoted as... .

[0121] The system uses the finite difference method to approximately calculate continuous differential operators on a discrete grid. Let the coordinates of the discrete grid be... , respectively corresponding to the reference coordinate system To simplify the formula, assume the mesh is in and The spacing in all directions is per unit length.

[0122] Gradient calculation: The gradient calculation unit calculates the final potential field. Perform gradient calculation at grid points gradient It is a two-dimensional vector, and the calculation uses the central difference method:

[0123] ;

[0124] in, This indicates that the final potential field is at the grid point. The value at that point. For points on the boundary, the forward or backward difference method can be used for calculation, and the output of this calculation is the gradient vector field. .

[0125] Divergence calculation: Divergence calculation element for energy flow field Perform divergence calculations at grid points. At, divergence It is a scalar, and the calculation also uses the central difference method:

[0126] ;

[0127] in, and These are energy flow fields At grid points place and The components, the output of the calculation is a divergence scalar field. .

[0128] Curl calculation: The curl calculation unit calculates the energy flow field. Performing curl calculations, in the two-dimensional case, curl is a scalar that characterizes the field in... The vortex component in the axial direction at the grid points Curl The calculation uses the central difference method:

[0129] ;

[0130] The output of this calculation is a curl scalar field. .

[0131] Through the above steps, the field feature analysis module generates the final potential field for each input time. It calculates and outputs three characteristic fields with clear physical and geometric meanings in real time: gradient field divergence field and curl field These feature fields will be passed to the generative audiovisual mapping engine as control signals.

[0132] See attached document Figure 5 The generative audiovisual mapping engine executes the generative mapping of this invention, receiving the final potential field from the field feature analysis module. gradient field divergence field Spool field Based on a pre-configured and modifiable set of mapping functions, the values ​​of these feature fields are converted in real time into specific control commands for each device in the stage effects execution device set.

[0133] The visual content generator internally implements a visual mapping function, which converts the input field feature data into control parameters that drive the real-time rendering algorithm in the video rendering server. These control parameters are then encapsulated into control messages (e.g., as shader uniform variables or sent via network protocols) and sent to the video rendering server. The following are specific examples of visual mapping functions:

[0134] Particle emissivity mapping: at each grid point in the audience reference coordinate system Above, the emission rate of the particle system From the final potential field The value is determined by a linear function:

[0135] ;in, and These are preset minimum and maximum particle emission rates. Particle emission rate mapping ensures that the higher the energy of the viewer, the denser the visual particles produced.

[0136] Particle initial velocity mapping: at grid points The initial velocity vector of the particle emitted at point The energy flow field at that point The value is determined by the scaling function:

[0137] ;in, It is a preset velocity scaling factor. The initial velocity mapping of the particles makes the initial motion direction and speed of the particles consistent with the actual flow trend of the audience's energy. In another embodiment, a gradient field can also be used. This drives the particle's velocity, representing the direction of potential energy increase.

[0138] Particle color and size mapping: Particle color and size From divergence field The value is determined, and the system presets two color values. and and two size values and Mapping is performed using a linear interpolation function:

[0139] ;

[0140] ;

[0141] in, Indicates in and The ratio between them is linear interpolation, The function will reduce the divergence value from its predicted range. Normalizing to a closed interval, this mapping causes the energy source (positive divergence) and sink (negative divergence) to exhibit different colors and sizes.

[0142] Particle force field mapping: There exists a force field in the rendered scene that acts on all particles. The force field is composed of the curl field. It is determined that for any position in space... The particles, which are at the grid points The rotational force acting nearby is calculated using the following formula:

[0143] ;

[0144] in, Grid points The center coordinates, It is a force scaling factor. It is a two-dimensional vector The operator that rotates by 90 degrees, i.e. The force field mapping of the particle causes the particle to generate a corresponding rotational motion in the energy vortex region.

[0145] The spatial audio synthesizer internally implements a set of audio mapping functions. These functions convert the input field feature data into Open Sound Control (OSC) or Musical Instrument Digital Interface (MIDI) messages, which are then sent to the audio server to control the synthesizer's parameters. The following is a specific example of these audio mapping functions:

[0146] Spatial soundscape gain mapping: The system divides the audience seating area into multiple zones, each corresponding to one or a group of virtual sound sources on the audio server. The overall gain of zone kk... The final potential field in this region average Determined by a logarithmic function, matching the human ear's perception of loudness:

[0147] ;in, and These are the preset minimum and maximum gains.

[0148] Synthesizer tone brightness mapping: the brightness of a background synthesizer tone throughout the entire soundstage (passed through the cutoff frequency of a low-pass filter). (Control) is determined by the global average value of the entire gradient field mode. Sure:

[0149] ;

[0150] in, and These are preset minimum and maximum cutoff frequencies. This mapping makes the timbre brighter and more penetrating when the audience's energy distribution changes drastically (large gradient).

[0151] Reverb effect parameter mapping: wet / dry ratio parameter of global reverb effect The global average of the absolute values ​​of the divergence field Linear determination:

[0152] ;

[0153] in, and These are the preset minimum and maximum wet / dry ratios. This mapping enhances the spatial sense of the sound when the overall energy field exhibits more explosive or converging dynamics.

[0154] Acoustic phase modulation rate mapping: for region The modulation rate of the associated sound source and its stereo phase. The average value of the absolute value of the curl field in this region Linear determination:

[0155] ;

[0156] in, It is the basic modulation rate. It is a rate scaling factor, which makes the sound appear to rotate or oscillate when energy vortices occur in the region.

[0157] See attached document Figure 6 Taking a specific application scenario as an example, system initialization and parameter loading: In system initialization and parameter loading, the system completes the startup, connection and configuration of all hardware and software, and loads all the static parameters required for operation, preparing for subsequent real-time data processing.

[0158] First, the central processing server starts up and loads its operating system and the system software described in this invention. The system software polls and establishes network communication connections with all online devices in the multimodal sensing device set and the stage effect execution device set through the data exchange network. The system software checks and confirms that the status of the thermal imaging array, visible light camera array, microphone array, motion capture device, lighting control device, audio server and video rendering server are all normal.

[0159] Next, the central processing server locates and reads one or more pre-defined configuration files from its local non-volatile storage device. These configuration files are stored in XML (Extensible Markup Language) or JSON (JavaScript Object Notation) format, defining all the static parameters required for system operation, including at least:

[0160] For the audience state acquisition unit: a perspective transformation matrix for mapping the image coordinates of each camera to a unified reference coordinate system; a network time protocol server address for time alignment; and upper and lower limit thresholds for normalizing each data field, including a lower temperature limit. and upper limit Maximum speed of motion lower sound pressure level and upper limit .

[0161] For the performer intent capture unit: an affine transformation matrix used to transform the local coordinate system of the motion capture equipment to the global reference coordinate system of the stage. ; Data on the mass ratio of each body part used to calculate the body's center of mass.

[0162] For the intent and potential field construction engine: the basic weight coefficients used for the generation of the basic potential field fusion. , , ; multiple standard pose template sequences stored in the pose library for intent classification; dynamic time warping distance threshold for determining intent matching; specific parameters for constructing various topological constraint functions, such as the residual coefficient of the energy suppression band in cutting intents. .

[0163] For the field feature analysis module: the resolution of the discrete grid used to perform finite difference calculations.

[0164] For generative audiovisual mapping engines: the definitions of all visual and audio mapping functions and their related parameters, such as the upper and lower limits of particle emission rate. and Preset speed scaling factor Divergence normalized range parameter The preset minimum and maximum gain of the audio gain and The filter's preset minimum and maximum cutoff frequencies and ; and the network address (IP address) and port number used to communicate with the stage effects execution device set.

[0165] After loading all parameters, the system software passes these parameters to their corresponding software modules. For example, it sets the fusion weight coefficient to the basic potential field generator, loads the pose template library to the performer intent classifier, and sets the visual mapping parameters to the visual content generator.

[0166] After all parameters have been loaded and configured, the system enters standby mode. In this state, all modules have been initialized, and the system awaits the operator's command to start operation, or awaits the multimodal sensing module to begin transmitting the first frame of valid data.

[0167] Subsequently, data acquisition and parallel processing are performed, and the system enters a continuous real-time loop processing state. The central processing server processes the real-time data streams from the audience and performers through two parallel execution threads, and transmits the processing results.

[0168] Real-time processing of audience data stream: Within an execution thread, the audience status acquisition unit continuously performs the following operations:

[0169] Data frame synchronization acquisition: The audience status acquisition unit extracts the set of synchronized data frames that are closest in timestamp from its internal data buffer. The data set includes thermal imaging image frames, visible light video frames and audio data blocks.

[0170] Spatial alignment and data processing: The audience status acquisition unit applies the loaded perspective transformation matrix to map the pixel coordinates of thermal imaging image frames and visible light video frames onto a unified two-dimensional reference coordinate system grid. At the same time, it executes a preset dense optical flow algorithm on the visible light video frames to calculate the motion vector of each point on the grid; and executes a preset beamforming algorithm on the audio data blocks to calculate the sound pressure level of each point on the grid.

[0171] Data normalization: Audience status acquisition unit 1 uses the loaded normalization parameters ( , , , , The normalization calculations were performed on the obtained temperature field, the magnitude (velocity) of the motion vector field, and the sound pressure field, respectively, and their values ​​were mapped to a closed interval.

[0172] Data output: Normalized temperature scalar field Scalar field of motion speed scalar field of sound pressure Together with the unnormalized original motion vector field (For subsequent energy flow field calculations), it is encapsulated into a data packet with a unified timestamp and sent to the basic potential field generator.

[0173] In another execution thread running parallel to the audience data processing thread, the performer intent capture unit works in conjunction with the performer intent classifier to perform the following operations:

[0174] Pose data acquisition and transformation: The performer intent capture unit receives the latest raw pose data from the motion capture device, and applies the loaded affine transformation matrix. The coordinates of the joint points are transformed from the local coordinate system of the equipment to the global reference coordinate system of the stage.

[0175] Feature vector calculation: Based on the transformed coordinates, the performer intent capture unit calculates a high-dimensional pose feature vector for the current frame, which includes information such as the state of the end-effectors, the angles of major joints, and the position of the body's center of mass.

[0176] Intent matching and classification: The performer intent capture unit adds the newly calculated feature vector to a fixed-length sliding window to form a query sequence. The performer intent classifier 2 receives the query sequence and calculates its dynamic time warping (DTW) distance with each intent template sequence in the pose library in parallel.

[0177] Instruction Generation and Output: The classifier compares all calculated DTW distances with the loaded DTW distance threshold. If one or more distances are found to be less than the threshold, a successful match is determined, and the intent corresponding to the template with the highest matching degree (i.e., the smallest distance) is taken as the current recognition result. Subsequently, the system extracts the parameters required for the intent from the successfully matched portion of the query sequence (e.g., the coordinates of the action center of the contraction intent are taken from the average position of the performer's hands during the matching period). Finally, the system encapsulates the recognized instruction type and the extracted parameters into a structured topological constraint instruction. If no match is found, an empty instruction or an instruction indicating no operation is generated. This instruction is appended with a timestamp corresponding to the input data and sent to the topology constraint processor.

[0178] Through the parallel processing mechanism described above, the system ensures that the perception of the audience's state and the interpretation of the performer's intentions can be carried out simultaneously, and the processing results are logically synchronized.

[0179] The final potential field is then dynamically constructed by receiving audience state data packets and performer intention commands, and calculating the final potential field at the current moment accordingly. .

[0180] The following example illustrates this step: At a certain moment, the audience in section A of the audience reacts enthusiastically (e.g., cheers and applause), and at the same time, the performer makes a gesture that guides or focuses attention towards section A.

[0181] Basic potential field The generator for generating the fundamental potential field receives audience state data packets. Based on the scene settings, within the coordinate range corresponding to area A of the audience seating area, the sound pressure scalar field... and motion speed scalar field The value is relatively high (close to 1.0), while the value is lower in other regions.

[0182] The basic potential field generator applies the loaded weight coefficients (e.g., let...). The fundamental potential field is calculated using the following formula.

[0183] ;

[0184] Because within the coordinate range corresponding to area A, and The contribution of this is significant; therefore, the calculated fundamental potential field... A high-value area is formed in region A, while the values ​​are lower in other regions. The field objectively represents the energy distribution spontaneously generated by the audience.

[0185] Topological constraint functions Construction: In parallel, the topology constraint processor receives performer intent instructions. Based on the scene setting, the performer intention classifier identifies the performer's gestures as contraction commands, and the performer intention commands... The data structure contains: an instruction of type "Constrict", and a center of action. Let the geometric center coordinates of audience area A be the intensity. The radius of influence is 1.5. Let A be the radius of region A.

[0186] The topology constraint processor constructs the corresponding topology constraint function based on this instruction. Its mathematical form is:

[0187] ;

[0188] This function At coordinate point The function reaches a maximum value of 2.5 at region A and decays smoothly outwards in a Gaussian distribution. Outside region A, the value of the function is close to 1.0. The field objectively represents the performer's intention to mathematically adjust the stage space.

[0189] Final Field The calculation: The topology constraint processor will calculate the fundamental potential field. With topological constraint functions By performing point-by-point multiplication, the final potential field is obtained. :

[0190] ;

[0191] In the calculation: within region A, due to and The values ​​are all high, and their product is... In this region, a higher level of development was formed. The peak value, which realizes the performer's intention to enhance and focus the audience's existing energy.

[0192] In areas other than Zone A, due to The value itself is low, even if The value is close to 1.0, the product It remains at a low level.

[0193] Through this step, the system generates a scalar field. After the calculation is completed, the final potential field is... Together with the original motion vector field at that moment It is sent as data to field feature analysis and audiovisual effect generation.

[0194] Field Feature Analysis and Audiovisual Effect Generation: The system generates the final potential field. Mathematical analysis is performed, and the results are mapped to specific control instructions for the stage effects execution device set. The final potential field is continued. An energy peak scene was created in section A of the audience seating area.

[0195] Field feature analysis module for And perform the following calculations: energy flow field Calculation: By and By multiplying each point, we obtain the energy flow field. .

[0196] In the scenario instance, due to area A The value is very high, indicating the physical movement of the audience within this area. (For example, the body movements and arm waving caused by cheering) are amplified, forming a strong energy flow with a clear direction.

[0197] Gradient, divergence, and curl calculations: The field characteristic analysis module uses the finite difference method to calculate the gradient field in parallel. divergence field Spool field .

[0198] gradient field The edge of the energy peak in region A points outward, and its modulus reaches its maximum at the steepest slope.

[0199] If the movement of the audience in area A is divergent (e.g., waving from the center outwards), then the divergence field... A positive value area is presented in the center of Area A, marking this location as the "source" of energy.

[0200] If the motion of the audience includes a rotational component (e.g., a group swaying from side to side), then the spin field... These will exhibit non-zero values ​​in the corresponding regions. After calculation, these four feature fields... It is sent as a data packet to the generative audiovisual mapping engine.

[0201] Generative audiovisual mapping and command issuance: The generative audiovisual mapping engine receives the above feature field data packets and generates control commands based on the loaded mapping function set.

[0202] Control of the video rendering server: The visual content generator performs the following mappings:

[0203] The final force field The value is mapped to the emission rate of the particle system, which corresponds to the position on the large screen in area A. The value leads to the generation of a large number of particles.

[0204] energy flow field The value is mapped to the initial velocity of the particle. At the position corresponding to area A, the particle is thrown out in the same direction and speed as the energy flow of the audience, creating a visual energy burst effect.

[0205] divergence field The positive value is obtained through the normalization function. Mapped to the color and size of particles, the particles corresponding to area A are rendered with a preset, bright "source" color (e.g., white) and have a larger initial size.

[0206] The generated particle emission rate, velocity, color and other parameters are encapsulated into shader uniform variables or custom network data packets, and sent to the video rendering server through the data exchange network to perform real-time rendering.

[0207] Control of the audio server: The spatial audio synthesizer performs the following mappings:

[0208] Calculate the final potential field in region A The average value is then mapped to the gain of the virtual sound source associated with area A using a logarithmic function.

[0209] Calculate the entire gradient field The global average value of the modulus is relatively high due to the steep energy peak in region A. This high value is mapped to the higher cutoff frequency of the low-pass filter in the background synthesizer timbre, making the overall background music timbre brighter.

[0210] Monitoring divergence field in area A When the value exceeds a preset positive threshold, the system determines that an energy burst event has occurred and generates a one-time trigger message.

[0211] The calculated gain value, cutoff frequency parameter, and event trigger message are encapsulated into a data packet conforming to the OSC protocol specification and sent to the audio server through the data exchange network. The audio server then increases the loudness of the sound in area A, making the background timbre clearer, and plays the preset burst sound effect.

[0212] Control of lighting equipment: Calculation of the final potential field in area A. The average value is linearly mapped to the brightness and beam angle of a set of stage lights pointing to area A, with a higher average value corresponding to higher brightness and a smaller beam angle.

[0213] The generated brightness and angle values ​​are encoded into data packets using the Art-Net or sACN protocol, which are then broadcast or unicast to the lighting control equipment via a data exchange network. This drives the corresponding lighting fixtures to execute commands, creating a spotlight effect focused on area A of the audience seating area.

[0214] Through the above steps, the system transforms abstract field characteristics into stage presentation, completing a closed loop from data perception to artistic expression. After that, the system cycles and updates in real time.

[0215] Looping and Real-time Updates: After issuing a complete set of control commands, the system does not terminate but immediately returns to process the next set of real-time data, ensuring that the system can make a continuous, low-latency response to any changes in the audience's state and the performer's intentions.

[0216] Establishment and synchronization of processing cycle: The system's working cycle is not executed indefinitely, but is driven by the data output rate of the multimodal sensing module. In a preferred embodiment, the visible light camera array works at a rate of frames per second (FPS). The rate defines the basic processing cycle of the system, that is, the system must complete a complete calculation process approximately every 33.3 milliseconds. The audience status acquisition unit uses this rate as a reference to push new synchronization data packets to subsequent modules, thereby triggering a single execution of the entire processing chain.

[0217] Smoothing of control parameters: To avoid unnatural and abrupt changes in audiovisual effects caused by slight fluctuations or instantaneous changes in input data, the generative audiovisual mapping engine performs time-dimensional smoothing of the calculated target parameters before issuing control commands.

[0218] Specifically, for any control parameter (e.g., light brightness) or audio gain The system not only stores the target value it has currently calculated, but also... It also maintains the actual value sent in the previous frame. The control values ​​ultimately sent to the execution device The result is obtained through the following linear interpolation formula:

[0219] ;

[0220] in, It is a preset smoothing coefficient that takes values ​​in the interval (0, 1). The closer the value is to 1, the faster the system responds, but the more prone it is to jumps. The closer the value is to 0, the smoother the system response, but the perceived latency will increase accordingly.

[0221] Dynamic adjustment mechanism for runtime parameters: The system described in this invention supports dynamic adjustment of key system parameters without interrupting the runtime loop. At the beginning of each loop, the central processing server checks specific memory flags or listens on a dedicated network port.

[0222] Operators can send commands to the network port through a separate graphical user interface (GUI) or console to modify system parameters stored in memory. These dynamically adjusted parameters include, but are not limited to:

[0223] Basic weight coefficients generated by the fusion of basic potential fields , , This involves real-time adjustments to the relative importance of different audience data modalities. Parameters in the generative audiovisual mapping function set can be used, for example, to change the mapping table from divergence field to color, or to adjust the intensity of the curl field's influence on the phase modulation rate. .

[0224] Control parameter smoothing coefficient When the system detects a parameter update command, it will immediately use the new parameter values ​​in subsequent calculations, thereby enabling real-time intervention and fine-tuning of the system's behavior.

[0225] Through the above-mentioned cyclical and real-time update mechanism, the system described in this invention, as a whole, manifests as a dynamic system with generative artistic response.

Claims

1. An interactive stage audiovisual presentation system incorporating audience feedback, characterized in that, include: A multimodal sensing device set is used to collect audience state data and performer posture data; A set of stage effects execution equipment used to present audiovisual effects; And, a software module configured in the central processing server, the software module comprising: The multimodal sensing module is used to receive and preprocess audience state data and performer posture data collected by the multimodal sensing device set; An intent and potential field construction engine is used to generate a basic potential field based on the audience state data, identify topological constraint instructions based on the performer posture data, construct a topological constraint function based on the topological constraint instructions, and then combine the topological constraint function with the basic potential field to calculate the final potential field. The field feature analysis module is used to perform mathematical operations on the final potential field and extract multiple feature fields. A generative audiovisual mapping engine is used to send the multiple feature field maps to the control instructions of the stage effects execution device set.

2. The interactive stage audiovisual presentation system integrating audience feedback as described in claim 1, characterized in that... The multimodal sensing device set includes a thermal imaging array, a visible light camera array, a microphone array, and a motion capture device; The stage effects execution equipment set includes lighting control equipment, audio servers, and video rendering servers.

3. The interactive stage audiovisual presentation system integrating audience feedback as described in claim 1, characterized in that, The intent and potential field construction engine includes: A basic potential field generator is used to generate the basic potential field based on the audience state data; A performer intent classifier is used to identify the performer pose data as the topological constraint instructions; A topology constraint processor is used to construct the topology constraint function according to the topology constraint instruction, and multiply the topology constraint function with the basic potential field point by point to calculate the final potential field.

4. The interactive stage audiovisual presentation system integrating audience feedback as described in claim 1, characterized in that, The multimodal sensing module includes: an audience state acquisition unit, used to receive raw data from a thermal imaging array, a visible light camera array, and a microphone array, and to perform spatial alignment, temporal alignment, and normalization processing on the raw data, outputting a temperature scalar field, a motion vector field, and a sound pressure scalar field as the audience state data.

5. The interactive stage audiovisual presentation system integrating audience feedback as described in claim 3, characterized in that, The performer intention classifier identifies and outputs the topological constraint instructions by matching real-time performer posture data sequences with templates in a preset posture library using a dynamic time warping algorithm.

6. The interactive stage audiovisual presentation system integrating audience feedback according to claim 3, characterized in that, The basic potential field generator uses a preset linear weighted fusion formula to fuse the temperature scalar field, motion velocity scalar field, and sound pressure scalar field in the audience state data into the basic potential field.

7. The interactive stage audiovisual presentation system integrating audience feedback as described in claim 3, characterized in that, The topology constraint processor selects different mathematical forms to construct the topology constraint function according to the type of the topology constraint instruction.

8. The interactive stage audiovisual presentation system integrating audience feedback according to claim 1, characterized in that, The field feature analysis module: An energy flow field is constructed by multiplying the final potential field point by point with the original motion vector field; Then, by performing differential geometric operations on the final potential field and the energy flow field, the gradient field, divergence field and curl field are calculated and output as the multiple characteristic fields.

9. The interactive stage audiovisual presentation system integrating audience feedback according to claim 1, characterized in that, The generative audiovisual mapping engine includes a visual content generator, which maps the multiple feature fields into a set of visual parameters through a preset visual mapping function. The visual parameters are encapsulated as part of the control instructions and sent to the video rendering server to control the execution of the real-time rendering algorithm. The visual parameters include: the emission rate of the particle system, the initial velocity of the particle system, the color of the particle system, the size of the particle system, and the force field parameters acting on the particle system in the real-time rendering algorithm.

10. The interactive stage audiovisual presentation system integrating audience feedback according to claim 1, characterized in that, The generative audiovisual mapping engine includes a spatial audio synthesizer, which maps the multiple feature fields into audio parameters through a preset audio mapping function. The audio parameters are encapsulated as part of the control instructions and sent to the audio server to control the synthesis and processing of audio. The audio parameters include: spatial soundscape gain, synthesizer timbre filter cutoff frequency, reverberation ratio (dry / wet) and pan modulation rate.

Images