An exhibition hall audio-visual experience display method and system based on a multi-modal AI model drive

Abstract

The application relates to a kind of exhibition hall audio-visual experience display method and system based on multi-modal AI model driving, belong to intelligent exhibition hall control field.Therein, the method includes inputting original signal flow for space-time synchronous pretreatment, obtaining audience multi-modal observation sequence;Variational inference is carried out based on audience multi-modal observation sequence to represent the collective state of audience, and the collective state representation is output;Based on the collective state representation and the current scene latent variable, the evolution driving force calculation on experience manifold is carried out, and the potential energy gradient is obtained;The scene latent variable is updated based on the potential energy gradient and multi-modal parameter decoding is carried out, and the updated scene latent variable and decoded multi-modal parameter set are output;Based on the multi-modal parameter set, cross-modal phase coherence constraint and final rendering output are carried out.The application realizes the evolution of driving scene latent variable along the potential energy gradient, thereby realizing the adaptive decoding and synchronous output of exhibition hall panoramic texture, spatial audio, light and vibration parameters.

Description

Technical Field

[0001] This invention belongs to the field of intelligent exhibition hall control technology, specifically relating to a method and system for displaying audio-visual experiences in exhibition halls based on a multimodal AI model. Background Technology

[0002] Currently, immersive audio-visual experience systems in exhibition halls mainly follow three technical approaches: The first is a timeline-based linear editing and control system, which pre-programs video, audio, lighting, and vibration into a fixed timeline script, relying on manual arrangement. While stable and reliable, this system completely lacks the ability to respond to real-time audience behavior, resulting in a monotonous experience for each exhibition and failing to cater to the diverse interests of different age groups and cultural backgrounds. The second is an interactive triggering system based on single-modal perception, such as using infrared cameras to detect audience positions and trigger content switching on nearby screens, or using microphone arrays to detect applause intensity and change volume. While such systems possess basic responsiveness... While powerful, these technologies disrupt the synergy of multimodal perception, often resulting in abrupt changes in sound and imagery due to lights turning off when people leave or location shifts. Furthermore, they fail to capture the continuous evolution of group attention. Thirdly, the recently emerging deep learning-based emotion computing or recommendation systems attempt to infer audience preferences and push relevant content through facial expression recognition or EEG devices. However, these methods are severely constrained in open exhibition environments by lighting, occlusion, and wearing comfort, leading to high deployment costs. They can only output discrete next-scene decisions and cannot smoothly manipulate continuous multidimensional physical parameters (such as panoramic texture deformation, sound field spatial vectors, and color temperature gradient slopes). Therefore, existing technologies generally lack continuous probabilistic modeling of the joint attention field of a group, and even more so, they lack a complete closed-loop framework that directly maps heterogeneous sensory feedback into the evolutionary driving force of latent variables in high-dimensional physical scenes. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, this invention provides a method and system for displaying audio-visual experiences in exhibition halls based on a multimodal AI model.

[0004] The objective of this invention can be achieved through the following technical solutions: A method for showcasing the audiovisual experience of an exhibition hall based on a multimodal AI model, the implementation of which includes the following steps: Step S1: Input the native signal stream in the physical space, specifically including the three-dimensional coordinate sequence of skeletal joints, the normalized pressure value grid, and the explicit semantic feedback from the audience. The explicit semantic feedback is submitted by the audience after scanning a QR code to access the local spatial network via a mobile terminal. Perform spatiotemporal synchronization preprocessing on the native signal stream to obtain a spatiotemporally consistent multimodal observation sequence of the audience. ; Step S2: Based on the audience multimodal observation sequence Perform variational inference of the collective state representation of the audience, and output the collective state representation. ; Step S3: Based on the collective state representation and current scene latent variables The evolutionary driving force on the experiential manifold M is calculated to obtain the potential energy gradient. The latent variables in the scenario It originates from the previous iteration of the next step, with the initial value being the implicit code of the default scene; Step S4: Based on the potential energy gradient Update the hidden variables of the scenario Multimodal parameter decoding is performed, and the updated scene latent variables are output. and the decoded multimodal parameter set ; Step S5: Based on the multimodal parameter set Perform cross-modal phase coherence constraints and final rendering output.

[0005] Preferably, the spatiotemporal synchronization preprocessing in step S1 specifically includes: Time alignment: Using SMPTE LTC linear timecode as the global reference, the timestamps of depth camera frames, pressure matrix sampling and mobile events are uniformly mapped to the same global clock domain. Interpolation resampling is performed using the system clock synchronized by PTP, so that all data streams are reduced to a unified discrete time step. Spatial registration: Transform the skeletal joints in each camera coordinate system to a unified exhibition hall world coordinate system W; Structured encapsulation: At each time step t, the skeletal sequence fragment of the nth viewer is encapsulated. Pressure distribution matrix The set of mobile semantic feedback vectors accumulated within the same step Packaged into the audience multimodal observation sequence .

[0006] Preferably, the variational inference of the collective state representation of the audience in step S2 specifically involves: A variational state estimator based on a combination of temporal graph attention and variational autoencoder is constructed, comprising three serial modules: a spatial interaction encoder, an attention field generator, and an interest topic inferencer. The spatial interaction encoder is used to process the skeletal sequence fragments of each viewer. Individual motion features are obtained by inputting each feature into a shared spatiotemporal graph convolutional network. Simultaneously, using individual locations as nodes and their orientation and distance as edge features, a fully connected spatial graph is constructed. Information is propagated through a graph attention layer, outputting social context-enhanced features for each viewer. The attention field generator enhances features with all of the aforementioned social contexts. and the corresponding head orientation unit vector Given the condition, normalized flow is used to estimate the viewer's line of sight at any point x in the entire space. Azimuth of the viewer's line of sight probability density ,in Given the hidden state prior of the previous time step, this probability density The interest-topic inference engine uses an adaptive kernel density estimate, with each viewer's position as the kernel and bandwidth related to their attention uncertainty, as a weighted mixture. This inference engine combines individual enhanced features with the mobile semantic feedback vector set. Input a multilayer perceptron and output the posterior parameters of the collective attention intensity of the audience to K latent topics. The latent topic is an abstract semantic category, which the inferencer aligns to the current audience signal through online variational updates; the collective state representation is finally output through three serial modules. ,in for The parameterized vector.

[0007] Preferably, the calculation of the evolutionary driving force in step S3 specifically includes: Before system deployment, a multimodal variational autoencoder is trained offline, whose latent space is the experience manifold M. The encoder of the multimodal variational autoencoder compresses the multimodal data of the complete scene into latent variables z, and the decoder is used to reconstruct the parameters from the latent variables z. During training, an expected attention prediction head is added to ensure that for any z, the network outputs the probability distribution of audience attention that the scene may cause under ideal conditions. ; The evolutionary potential energy was calculated. Mathematically described ,in, To balance the parameters, To experience the prior probability density of a manifold, For information content; Using automatic differentiation, the latent variables in the scenario The above calculation of the evolutionary potential energy The potential energy gradient is obtained by considering the gradient vector of the latent variable z. .

[0008] Preferably, the latent variables in step S4 are... The update is as follows: Based on the potential energy gradient Performing a state update, mathematically described as... ,in, Let dt be the mobility and dt be the time derivative. The diffusion coefficient is... The increment is a multivariable standard Wiener process; Euler-Maruyama discretization is used for easier implementation in discrete time steps. ,in, For time step, This is a standard Gaussian white noise vector.

[0009] Preferably, the multimodal parameter decoding in step S4 specifically involves: The decoder of the pre-trained multimodal variational autoencoder is invoked to generate all modal control parameters required for the current time step at once, and the resulting set is the multimodal parameter set. ,in A is the geometrically corrected panoramic texture map required for projection; A is the list of audio objects; L is the light DMX control vector; and F is the floor vibration feedback mode.

[0010] Preferably, the cross-modal phase coherence constraint in step S5 specifically includes: Will Using the current frame and combining it with the texture of the previous frame, the GPU-accelerated Farneback optical flow algorithm is used to calculate the instantaneous motion vector of each pixel in screen space, thus obtaining the visual optical flow velocity field. ; Extract the 3D trajectory of each active audio object from A, map it to screen coordinates using a projection matrix to obtain the screen position, calculate the instantaneous velocity using numerical difference, and interpolate according to the sound intensity contribution weight of the object at each position on the screen to generate a full-field sound image velocity field. ; Calculate synchronization loss Mathematically described ,in For the spatial domain of the projection screen, For the time step, based on the synchronization loss Perform feedforward compensation; The audio parameters after feedforward compensation and the multimodal parameter set They are submitted together to each subsystem to complete the final rendering output.

[0011] A multimodal AI model-driven exhibition hall audio-visual experience display system is used to execute the multimodal AI model-driven exhibition hall audio-visual experience display method described above, including a data processing module, a variational inference module, an evolutionary driving force calculation module, an update and decoding module, and a rendering output module; The data processing module is used to input the native signal stream in the physical space, specifically including the three-dimensional coordinate sequence of skeletal joints, the normalized pressure value grid, and the explicit semantic feedback from the audience. The explicit semantic feedback is submitted by the audience after scanning a QR code to access the local spatial network via a mobile terminal. The native signal stream is then subjected to spatiotemporal synchronized preprocessing to obtain a spatiotemporally consistent multimodal observation sequence of the audience. ; The variational inference module is used to perform analysis based on the audience multimodal observation sequence. Perform variational inference of the collective state representation of the audience, and output the collective state representation. ; The evolutionary driving force calculation module is used to calculate based on the collective state representation. and current scene latent variables The evolutionary driving force on the experiential manifold M is calculated to obtain the potential energy gradient. The latent variables in the scenario It originates from the previous iteration of the next step, with the initial value being the implicit code of the default scene; The update and decoding module is used to update based on the potential energy gradient. Update the hidden variables of the scenario Multimodal parameter decoding is performed, and the updated scene latent variables are output. and the decoded multimodal parameter set ; The rendering output module is used to output data based on the multimodal parameter set. Perform cross-modal phase coherence constraints and final rendering output.

[0012] The beneficial effects of this invention are as follows: (1) By constructing a collective state representation that combines temporal graph attention and variational inference, continuous probabilistic modeling of the spatial distribution of audience attention and interest tendencies is realized, replacing the traditional discrete trigger response mechanism.

[0013] (2) By introducing evolutionary potential energy combining KL divergence and novelty potential in the latent space and updating the latent variables of the scene, the system achieves a dynamic balance between real-time tracking of audience preferences and active exploration of novel scenes.

[0014] (3) By introducing a synchronous loss calculation and feedforward compensation mechanism based on the comparison of visual optical flow and sound image velocity field, the phase consistency correction between dynamic visual texture movement and spatial audio object drift is realized, reducing the discomfort caused by sound and image misalignment.

[0015] (4) By integrating the coordinates of skeletal joints, ground pressure distribution and semantic feedback from mobile terminals as a unified driving signal, the coordinated and continuous control of multimodal outputs such as panoramic texture, spatial audio parameters, lighting and vibration feedback is realized without the need for manual segmentation and arrangement. Attached Figure Description

[0016] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0017] Figure 1 This is a flowchart illustrating the steps of a method for displaying an audio-visual experience in an exhibition hall based on a multimodal AI model, according to the present invention. Detailed Implementation

[0018] To better understand the invention, various aspects of the invention will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of the invention and are not intended to limit the scope of the invention in any way. Throughout the specification, the expression "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the terms "approximately," "about," and similar terms are used as expressions of approximation, not as expressions of degree, and are intended to describe inherent deviations in measured or calculated values ​​that will be recognized by those skilled in the art. Furthermore, the order in which the steps are described in this invention does not necessarily indicate the order in which these steps occur in actual operation, unless otherwise expressly defined or deduced from the context.

[0019] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of the invention, the word "may" is used to mean "one or more embodiments of the invention." And the term "exemplary" is intended to refer to examples or illustrations.

[0020] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formalized sense.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] Example 1: Please see Figure 1 A method for showcasing the audiovisual experience of an exhibition hall, driven by a multimodal AI model, includes: Step S1: Input the native signal stream in the physical space, specifically including the 3D coordinate sequence of skeletal joints output by the depth camera at the top of the exhibition hall, the normalized pressure value grid output by the ground pressure sensing matrix, and the explicit semantic feedback from the audience. The explicit semantic feedback is submitted by the audience after scanning a QR code to access the local spatial network via a mobile terminal, such as tags of interest, sentiment symbols, or voting options. Perform spatiotemporal synchronization preprocessing on the native signal stream to obtain a spatiotemporally consistent multimodal observation sequence of the audience. ; Step S2: Based on the audience multimodal observation sequence Perform variational inference of the collective state representation of the audience, and output the collective state representation. High-dimensional, heterogeneous primitive observations Compress and sublimate into a probabilistic collective state representation This representation simultaneously encodes the joint attention spatial distribution of the audience group and the distribution information of potential interest topics, and strictly maintains the form of a continuous probability distribution to support the subsequent continuous evolution mechanism. Step S3: Based on the collective state representation and current scene latent variables The evolutionary driving force on the experiential manifold M is calculated to obtain the potential energy gradient. The latent variables in the scenario It originates from the previous iteration of the next step, with the initial value being the implicit code of the default scene; Step S4: Based on the potential energy gradient Update the hidden variables of the scenario Multimodal parameter decoding is performed, and the updated scene latent variables are output. and the decoded multimodal parameter set ; Step S5: Based on the multimodal parameter set Cross-modal phase coherence constraints and final rendering output are performed. The visual and audio parameters generated in the previous step are then finely synchronized and corrected in real time with low latency to ensure that the movement of visual objects is consistent with the spatial movement of corresponding audio objects in the final physical rendering stage, thus eliminating dynamic audio-visual misalignment.

[0023] In this embodiment, the spatiotemporal synchronization preprocessing specifically includes: Time alignment: Using SMPTE LTC linear timecode as the global reference, the timestamps of depth camera frames, pressure matrix sampling and mobile events are uniformly mapped to the same global clock domain. Interpolation resampling is performed using the system clock synchronized by PTP, so that all data streams are reduced to a unified discrete time step. Spatial registration: Transform the skeletal joints in each camera coordinate system to a unified exhibition hall world coordinate system W; Structured encapsulation: At each time step t, the skeletal sequence fragment of the nth viewer is encapsulated. Pressure distribution matrix The set of mobile semantic feedback vectors accumulated within the same step Packaged into the audience multimodal observation sequence .

[0024] In this embodiment, the variational inference of the collective state representation of the audience specifically involves: A variational state estimator based on a combination of temporal graph attention and variational autoencoder is constructed, comprising three serial modules: a spatial interaction encoder, an attention field generator, and an interest topic inferencer. The spatial interaction encoder is used to process the skeletal sequence fragments of each viewer. Individual motion features are obtained by inputting each feature into a shared spatiotemporal graph convolutional network. Simultaneously, using individual locations as nodes and their orientation and distance as edge features, a fully connected spatial graph is constructed. Information is propagated through a graph attention layer, outputting social context-enhanced features for each viewer. The attention field generator enhances features with all of the aforementioned social contexts. and the corresponding head orientation unit vector Given the condition, normalized flow is used to estimate the viewer's line of sight at any point x in the entire space. Azimuth of the viewer's line of sight probability density ,in Given the hidden state prior of the previous time step (the initial step uses a standard normal prior), this probability density... The interest-topic inference engine uses an adaptive kernel density estimate, with each viewer's position as the kernel and bandwidth related to their attention uncertainty, as a weighted mixture. This inference engine combines individual enhanced features with the mobile semantic feedback vector set. Input a multilayer perceptron and output the posterior parameters of the collective attention intensity of the audience to K latent topics. The latent topics are abstract semantic categories obtained through unsupervised clustering of massive panoramic video-audio scenes during the system's offline phase. The inferencer aligns these categories to the current viewer signal through online variational updates. Finally, the collective state representation is output through three serial modules. ,in for The parameterized vectors (mean, covariance, and weights of each Gaussian mixture component) are used to compactly describe the attention distribution.

[0025] In this embodiment, the calculation of the evolutionary driving force specifically involves: S301: Before system deployment, a multimodal variational autoencoder is trained offline. Its latent space is the experience manifold M. The encoder of the multimodal variational autoencoder compresses the multimodal data of the complete scene (such as the latent representation of panoramic video frames, spatial parameterization of audio objects, light curves, vibration modes, etc.) into latent variables z. The decoder is used to reconstruct the parameters from the latent variables z. During training, an expected attention prediction head is added to enable the network to output the probability distribution of audience attention that the scene may cause under ideal conditions for any z. (This distribution has been internalized during the training phase and does not require online optimization.) S302: Calculate the evolutionary potential energy Mathematically described ,in, To balance the parameters and adjust the settings between adapting to the audience and maintaining content familiarity, To experience the prior probability density of the manifold (determined by the scene distribution of the training set). That is, information content; common and ordinary z has high information content. Low novelty potential; rare and unusual z has high novelty potential. The negative sign means that high novelty will reduce the total potential energy, thereby attracting the system to explore novel regions. S303: Utilizing automatic differentiation, in the context of latent variables The above calculation of the evolutionary potential energy The potential energy gradient is obtained by considering the gradient vector of the latent variable z. .

[0026] In this embodiment, the scene latent variables The update is as follows: Based on the potential energy gradient Performing a state update, mathematically described as... ,in, Mobility, representing the speed at which changes in the audience's state translate into scene evolution. The larger the value, the more sensitively the system follows the audience; dt is the time derivative. The diffusion coefficient is... The increment is a multivariable standard Wiener process; Euler-Maruyama discretization is used for easier implementation in discrete time steps. ,in, For time step, It is a standard Gaussian white noise vector; The multimodal parameter decoding specifically involves: The decoder of the pre-trained multimodal variational autoencoder is invoked to generate all modal control parameters required for the current time step at once, and the resulting set is the multimodal parameter set. ,in A is the geometrically corrected panoramic texture map required for projection (streamed to the projection blending processor, replacing the existing texture cache); A is a list of audio objects, each containing 3D coordinates, directionality, attenuation radius, and amplitude envelope, which can be directly input into the spatial audio rendering engine; L is the light DMX control vector, including color temperature, intensity, fading time, etc.; F is the floor vibration feedback mode, including frequency, amplitude, and duration envelope.

[0027] In this embodiment, the cross-modal phase coherence constraint specifically refers to: S501: Will Using the current frame and combining it with the texture of the previous frame, the GPU-accelerated Farneback optical flow algorithm is used to calculate the instantaneous motion vector of each pixel in screen space, thus obtaining the visual optical flow velocity field. ; S502: Extract the 3D trajectory of each active audio object from A, map it to screen coordinates using a projection matrix to obtain the screen position, calculate the instantaneous velocity using numerical difference, and interpolate according to the sound intensity contribution weight of the object at each position on the screen to generate a full-field sound image velocity field. ; S503: Calculate synchronization loss Mathematically described ,in For the spatial domain of the projection screen, For the time step, based on the synchronization loss Feedforward compensation is performed: In the playback engine, a tiny, dynamically adjusted variable delay line is introduced for each audio object, with the delay amount determined by the amplitude of the local velocity difference. Specifically, for each audio object k, if it is within its main sound image coverage area... and If the angle between the two objects is too large or the rate difference exceeds the threshold, the trigger phase of the audio object is adjusted by a delay proportional to the difference, or its spatial position interpolation slope is finely adjusted to match the sound image movement rate with the local visual motion. S504: The audio parameters after the feedforward compensation and the multimodal parameter set They are submitted together to each subsystem to complete the final rendering output.

[0028] Example 2: An exhibition hall audio-visual experience display system driven by a multimodal AI model includes a data processing module, a variational inference module, an evolutionary driving force calculation module, an update and decoding module, and a rendering output module. The data processing module is used to input the native signal stream in the physical space, specifically including the three-dimensional coordinate sequence of skeletal joints output by the depth camera at the top of the exhibition hall, the normalized pressure value grid output by the ground pressure sensing matrix, and explicit semantic feedback from the audience. This explicit semantic feedback is submitted by the audience after scanning a QR code to access the local spatial network via a mobile terminal, such as tags of interest, sentiment symbols, or voting options. The native signal stream undergoes spatiotemporal synchronized preprocessing to obtain a spatiotemporally consistent multimodal observation sequence of the audience. ; The variational inference module is used to perform analysis based on the audience multimodal observation sequence. Perform variational inference of the collective state representation of the audience, and output the collective state representation. High-dimensional, heterogeneous primitive observations Compress and sublimate into a probabilistic collective state representation This representation simultaneously encodes the joint attention spatial distribution of the audience group and the distribution information of potential interest topics, and strictly maintains the form of a continuous probability distribution to support the subsequent continuous evolution mechanism. The evolutionary driving force calculation module is used to calculate based on the collective state representation. and current scene latent variables The evolutionary driving force on the experiential manifold M is calculated to obtain the potential energy gradient. The latent variables in the scenario It originates from the previous iteration of the next step, with the initial value being the implicit code of the default scene; The update and decoding module is used to update based on the potential energy gradient. Update the hidden variables of the scenario Multimodal parameter decoding is performed, and the updated scene latent variables are output. and the decoded multimodal parameter set ; The rendering output module is used to output data based on the multimodal parameter set. Cross-modal phase coherence constraints and final rendering output are performed. The visual and audio parameters generated in the previous step are then finely synchronized and corrected in real time with low latency to ensure that the movement of visual objects is consistent with the spatial movement of corresponding audio objects in the final physical rendering stage, thus eliminating dynamic audio-visual misalignment.

[0029] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for showcasing an audio-visual experience in exhibition halls based on a multimodal AI model, characterized in that, Includes the following steps: Step S1: Input the native signal stream in the physical space, specifically including the three-dimensional coordinate sequence of skeletal joints, the normalized pressure value grid, and the explicit semantic feedback of the audience. The explicit semantic feedback is submitted by the audience after scanning the code to access the local network of the space via a mobile terminal. By performing spatiotemporal synchronization preprocessing on the original signal stream, a spatiotemporally consistent multimodal observation sequence of the audience is obtained. ; Step S2: Based on the audience multimodal observation sequence Perform variational inference of the collective state representation of the audience, and output the collective state representation. ; Step S3: Based on the collective state representation and current scene latent variables The evolutionary driving force on the experiential manifold M is calculated to obtain the potential energy gradient. The latent variables in the scenario It originates from the previous iteration of the next step, with the initial value being the implicit code of the default scene; Step S4: Based on the potential energy gradient Update the hidden variables of the scenario Multimodal parameter decoding is performed, and the updated scene latent variables are output. and the decoded multimodal parameter set ; Step S5: Based on the multimodal parameter set Perform cross-modal phase coherence constraints and final rendering output.

2. The exhibition hall audio-visual experience display method based on multimodal AI model driven according to claim 1, characterized in that, The spatiotemporal synchronization preprocessing in step S1 specifically includes: Time alignment: Using SMPTE LTC linear timecode as the global reference, the timestamps of depth camera frames, pressure matrix sampling and mobile events are uniformly mapped to the same global clock domain. Interpolation resampling is performed using the system clock synchronized by PTP, so that all data streams are reduced to a unified discrete time step. Spatial registration: Transform the skeletal joints in each camera coordinate system to a unified exhibition hall world coordinate system W; Structured encapsulation: At each time step t, the skeletal sequence fragment of the nth viewer is encapsulated. Pressure distribution matrix The set of mobile semantic feedback vectors accumulated within the same step Packaged into the audience multimodal observation sequence .

3. The exhibition hall audio-visual experience display method based on multimodal AI model driven according to claim 2, characterized in that, The variational inference of the collective state representation of the audience in step S2 is specifically as follows: A variational state estimator based on a combination of temporal graph attention and variational autoencoder is constructed, comprising three serial modules: a spatial interaction encoder, an attention field generator, and an interest topic inferencer. The spatial interaction encoder is used to process the skeletal sequence fragments of each viewer. Individual motion features are obtained by inputting each feature into a shared spatiotemporal graph convolutional network. Simultaneously, using individual locations as nodes and their orientation and distance as edge features, a fully connected spatial graph is constructed. Information is propagated through a graph attention layer, outputting social context-enhanced features for each viewer. The attention field generator enhances features with all of the aforementioned social contexts. and the corresponding head orientation unit vector Given the condition, normalized flow is used to estimate the viewer's line of sight at any point x in the entire space. Azimuth of the viewer's line of sight probability density ,in Given the hidden state prior of the previous time step, this probability density The interest-topic inference engine uses an adaptive kernel density estimate, with each viewer's position as the kernel and bandwidth related to their attention uncertainty, as a weighted mixture. This inference engine combines individual enhanced features with the mobile semantic feedback vector set. Input a multilayer perceptron and output the posterior parameters of the collective attention intensity of the audience to K latent topics. The latent topics are abstract semantic categories, which the inferencer aligns to the current audience signal through online variational updates; The collective state representation is ultimately output through three serial modules. ,in for The parameterized vector.

4. The exhibition hall audio-visual experience display method based on multimodal AI model driven according to claim 3, characterized in that, The calculation of the evolutionary driving force in step S3 is specifically as follows: Before system deployment, a multimodal variational autoencoder is trained offline, whose latent space is the experience manifold M. The encoder of the multimodal variational autoencoder compresses the multimodal data of the complete scene into latent variables z, and the decoder is used to reconstruct the parameters from the latent variables z. During training, an expected attention prediction head is added to ensure that for any z, the network outputs the probability distribution of audience attention that the scene may cause under ideal conditions. ; The evolutionary potential energy was calculated. Mathematically described ,in, To balance the parameters, To experience the prior probability density of a manifold, For information content; Using automatic differentiation, the latent variables in the scenario The above calculation of the evolutionary potential energy The potential energy gradient is obtained by considering the gradient vector of the latent variable z. .

5. The exhibition hall audio-visual experience display method based on multimodal AI model driven according to claim 4, characterized in that, The latent variables in step S4 The update is as follows: Based on the potential energy gradient Performing a state update, mathematically described as... ,in, Let dt be the mobility and dt be the time derivative. Where is the diffusion coefficient. The increment is a multivariable standard Wiener process; Euler-Maruyama discretization is used for easier implementation in discrete time steps. ,in, For time step, This is a standard Gaussian white noise vector.

6. The exhibition hall audio-visual experience display method based on multimodal AI model driven according to claim 5, characterized in that, The multimodal parameter decoding in step S4 specifically involves: The decoder of the pre-trained multimodal variational autoencoder is invoked to generate all modal control parameters required for the current time step at once, and the resulting set is the multimodal parameter set. ,in A is the geometrically corrected panoramic texture map required for projection; L is the audio object list; F is the light DMX control vector; and F is the floor vibration feedback mode.

7. The exhibition hall audio-visual experience display method based on multimodal AI model driven according to claim 6, characterized in that, The cross-modal phase coherence constraint in step S5 specifically refers to: Will Using the current frame and combining it with the texture of the previous frame, the GPU-accelerated Farneback optical flow algorithm is used to calculate the instantaneous motion vector of each pixel in screen space, thus obtaining the visual optical flow velocity field. ; Extract the 3D trajectory of each active audio object from A, map it to screen coordinates using a projection matrix to obtain the screen position, calculate the instantaneous velocity using numerical difference, and interpolate according to the sound intensity contribution weight of the object at each position on the screen to generate a full-field sound image velocity field. ; Calculate synchronization loss Mathematically described ,in For the spatial domain of the projection screen, For the time step, based on the synchronization loss Perform feedforward compensation; The audio parameters after feedforward compensation and the multimodal parameter set They are submitted together to each subsystem to complete the final rendering output.

8. A multimodal AI model-driven audio-visual experience display system for exhibition halls, characterized in that, The system is applied to the exhibition hall audio-visual experience display method based on multimodal AI model driven as described in any one of claims 1-7, including a data processing module, a variational inference module, an evolutionary driving force calculation module, an update and decoding module, and a rendering output module; The data processing module is used to input the native signal stream in the physical space, specifically including the three-dimensional coordinate sequence of skeletal joints, the normalized pressure value grid, and the explicit semantic feedback of the audience. The explicit semantic feedback is submitted by the audience after scanning a code to access the local network of the space via a mobile terminal. By performing spatiotemporal synchronization preprocessing on the original signal stream, a spatiotemporally consistent multimodal observation sequence of the audience is obtained. ; The variational inference module is used to perform analysis based on the audience multimodal observation sequence. Perform variational inference of the collective state representation of the audience, and output the collective state representation. ; The evolutionary driving force calculation module is used to calculate based on the collective state representation. and current scene latent variables The evolutionary driving force on the experiential manifold M is calculated to obtain the potential energy gradient. The latent variables in the scenario It originates from the previous iteration of the next step, with the initial value being the implicit code of the default scene; The update and decoding module is used to update based on the potential energy gradient. Update the hidden variables of the scenario Multimodal parameter decoding is performed, and the updated scene latent variables are output. and the decoded multimodal parameter set ; The rendering output module is used to output data based on the multimodal parameter set. Perform cross-modal phase coherence constraints and final rendering output.

Images