An interactive dome viewing system and method based on AI multi-modal data processing

Abstract

The application discloses an interactive dome screen viewing system and method based on AI multi-modal data processing, relates to the technical field of data processing, and comprises a background motion isolation module, a interactive window period dynamic adjustment module, a modal source tracking module, a dome screen change trend prediction module and a modal anti-interference filtering module.The background motion isolation module analyzes and processes dome screen video stream signals through an optical flow estimation process; the interactive window period dynamic adjustment module dynamically adjusts the response state time window of the system; the modal source tracking module dynamically evaluates the reliability of the sources of various modal signals; the dome screen change trend prediction module fuses a convolutional neural network (CNN) and a long short-term memory (LSTM) network to predict the future disturbance trend of the background of dome screen images; the modal anti-interference filtering module generates anti-interference samples based on a diffusion generative adversarial model training method; and the signal fusion module integrates the processing results from various modules.The application can effectively distinguish background visual stimulation from the real interactive intention of a user, and improves the stability and accuracy of interaction.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically an interactive dome theater viewing system and method based on AI multimodal data processing. Background Technology

[0002] A dome theater system is an advanced immersive multimedia display platform, typically used in large cinemas, museums, educational venues, and other environments. It displays content through a 360-degree immersive visual experience on a spherical screen, thereby enhancing the audience's sense of immersion and participation. With the advancement of technology, dome theater systems are no longer limited to visual presentation; they have also begun to introduce multimodal input methods such as voice, gestures, and head movements, allowing users to interact with the system naturally.

[0003] In a dome theater environment, due to the drastic changes in the projected content (such as explosions, transitions, and flashbacks) and the high degree of overlap with the user's actual actions / voice interactions, the system often experiences confusion on two levels:

[0004] If modal drift is falsely triggered, the system will treat changes in projected visual information as user behavior signals and switch modalities (such as switching from "gaze perception" to "voice input").

[0005] Modal semantic conflict triggering occurs when the system encounters inconsistent multimodal inputs during real-world interactions (such as a user's voice "fast forward" combined with a vague gesture) and incorrectly triggers the command. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing an interactive dome theater viewing system and method based on AI multimodal data processing, which solves the technical problems mentioned in the background.

[0007] To address the above technical problems, the following technical solution is adopted: This invention provides an interactive dome theater viewing system based on AI multimodal data processing, comprising:

[0008] The background motion isolation module analyzes and processes the dome video stream signal through optical flow estimation, extracts the spatiotemporal change characteristics of the current frame image, and calculates the background perturbation confidence level in conjunction with the background modeling process.

[0009] The interactive window period dynamic adjustment module is used to dynamically adjust the system's response state time window based on the temporal coordination between the user's body movement input signal and voice input signal.

[0010] The modal source tracking module is based on the Transformer neural network architecture. It performs feature vector correlation modeling on modal signals from different input channels through a multi-head attention mechanism and dynamically evaluates the credibility of each modal signal source.

[0011] The dome screen change trend prediction module integrates a convolutional neural network (CNN) and a long short-term memory network (LSTM) to predict the future perturbation trend of the dome screen image background.

[0012] A modal anti-interference filtering module generates anti-interference samples based on a diffusion adversarial generative model training method, and optimizes the robustness parameters of the filter function through a neural network;

[0013] The signal fusion module integrates the processing results from each module and comprehensively evaluates the background disturbance intensity, the reliability of the modal signal source, and the interactive window status.

[0014] The present invention also provides an interactive dome screen viewing method, comprising the following steps: collecting the user's motion input signal, voice input signal and visual attention information, and using the motion sensor array, directional voice receiving array and gaze trajectory tracking unit in the modal perception module to achieve synchronous collection;

[0015] Optical flow estimation and background modeling are performed on the dome video stream through the background motion isolation module, the spatiotemporal variation characteristics of the current image are extracted and the confidence of background perturbation is calculated;

[0016] Based on the user's body movements and voice input signals, the interaction window dynamic adjustment module dynamically adjusts the response time window of the interaction window according to the temporal characteristics of the body movement and voice input signals.

[0017] The modal source tracking module is used to perform feature vector correlation modeling for input signals of different modes and dynamically evaluate the credibility of each modal signal source, providing a basis for signal fusion;

[0018] The dome screen change trend prediction module integrates convolutional neural networks and long short-term memory networks to predict the future perturbation trend of the dome screen image background and outputs the future perturbation probability value.

[0019] An anti-interference sample set is generated by a modal anti-interference filtering module, and the robustness parameters of the filter are optimized by a neural network to filter out interference signals.

[0020] The processing results of each module are integrated through the signal fusion module to comprehensively evaluate the background disturbance intensity, modal signal source credibility, interactive window state, signal-to-noise ratio and disturbance prediction probability, and generate fused expression values.

[0021] The beneficial effects of this invention are:

[0022] 1. In this invention, through the collaborative work of the background motion isolation module and the dome screen change trend prediction module, the system can assess the confidence level of the current dome screen background disturbance in real time and predict its future change trend, thereby dynamically adjusting the processing sensitivity to user input signals. This mechanism significantly reduces the risk of modal drift caused by visual drastic changes such as projection screen explosion and flashing, enabling the system to effectively distinguish between "background visual stimuli" and "user's real interaction intentions", thus improving the stability and accuracy of the interaction.

[0023] 2. This invention, by introducing a dynamic adjustment module for the interaction window period and a modal source tracking module, can combine the time alignment of multimodal signals, the synchronization of actions and speech, and the credibility score of the signal source to achieve intelligent recognition and reasonable interpretation of asynchronous input. When there is a slight misalignment between the user's speech and actions, or a weak modality (such as a blurry gesture), the system can still accurately determine the user's interaction intention, thereby effectively reducing the occurrence rate of "false triggering of modal semantic conflicts".

[0024] 3. In this invention, through the modal anti-interference filtering module and the signal fusion module, the system introduces a diffusion adversarial generative model to generate modal interference samples for training, thereby improving the ability of the neural network filter to distinguish real input signals. The final fused output not only considers the interaction state and background disturbance intensity, but also combines modal credibility and filter signal-to-noise ratio to ensure that the command is triggered only when the fused expression value is higher than the set threshold, which significantly enhances the system's ability to resist false triggering, especially in environments with noise interference or screen turbulence, it can still maintain interaction accuracy. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] In the attached diagram:

[0027] Figure 1 This is a schematic diagram of the interactive dome viewing system of the present invention.

[0028] Figure 2 This is a flowchart illustrating the interactive dome screen viewing method of the present invention. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description of this invention.

[0030] Example

[0031] like Figure 1 As shown, an interactive dome theater viewing system based on AI multimodal data processing includes:

[0032] The background motion isolation module analyzes and processes the dome video stream signal through optical flow estimation, extracts the spatiotemporal change characteristics of the current frame image, and calculates the background perturbation confidence level in conjunction with the background modeling process.

[0033] The interactive window period dynamic adjustment module is used to dynamically adjust the system's response state time window based on the temporal coordination between the user's body movement input signal and voice input signal.

[0034] The modal source tracking module is based on the Transformer neural network architecture. It performs feature vector correlation modeling on modal signals from different input channels through a multi-head attention mechanism and dynamically evaluates the credibility of each modal signal source.

[0035] The dome screen change trend prediction module integrates a convolutional neural network (CNN) and a long short-term memory network (LSTM) to predict the future perturbation trend of the dome screen image background.

[0036] A modal anti-interference filtering module generates anti-interference samples based on a diffusion adversarial generative model training method, and optimizes the robustness parameters of the filter function through a neural network;

[0037] The signal fusion module integrates the processing results from each module and comprehensively evaluates the background disturbance intensity, the reliability of the modal signal source, and the interactive window status.

[0038] This invention, through modular design and the introduction of an AI deep learning architecture, not only solves the problem of traditional interactive systems being susceptible to interference in dome environments, but also has the following technical advantages: high accuracy of multimodal signal fusion, strong ability to automatically identify and avoid background disturbances, significantly improved overall system robustness and fault tolerance, dynamic adaptability, and suitability for deployment in complex dynamic audiovisual scenarios.

[0039] Furthermore, the working process of the background motion isolation module is as follows:

[0040] The optical flow estimation process is used to extract motion vector information of each pixel within the current image frame;

[0041] Specifically, optical flow estimation technology analyzes the motion between image frames based on the motion vector of each pixel in the image. This process analyzes each frame in the video stream, extracts the motion information of each pixel, and thus determines the visual changes of the current frame compared to the previous frame. This process can accurately capture the dynamic effects in the background caused by changes in scene or content, and provides input data for subsequent background modeling by calculating the motion vector of each pixel.

[0042] In practical applications, optical flow estimation does not rely solely on a single image frame, but rather derives an overall background motion trend by comparing multiple consecutive image frames. This method can capture the overall changes in the background within a short period of time, and is particularly effective in detecting the dynamic features of these background changes when the background content changes rapidly (such as explosions or transitions).

[0043] The background modeling process calculates the motion vector difference based on the expected position of the predicted pixel and obtains the perturbation intensity, and then further generates the background perturbation confidence value;

[0044] Specifically, background modeling further refines the background perturbation intensity based on optical flow estimation. By calculating the difference between the expected position and the actual movement position of the predicted pixel, the intensity of possible perturbations in the background can be assessed. Background perturbations are usually manifested as rapid changes in the image, sudden changes in color or brightness, or sudden movements in certain areas. The system generates the intensity value of the background perturbation by calculating the difference between the motion vector of each pixel and the expected position.

[0045] In the core part of this process, the background perturbation confidence value (C) t The data is generated and used as a key basis for subsequent processing. The higher the confidence value, the stronger the background disturbance. The system should treat the current input modal signal more cautiously. In a dome screen environment, drastic changes in visual content may cause the system to misjudge user behavior. Especially when the background content changes drastically, the user may trigger irrelevant interactive behaviors due to natural reactions. Through accurate background disturbance judgment, the system can effectively distinguish between drastic changes in the background and the user's actual input.

[0046] The confidence level value serves as one of the bases for the signal fusion module to determine the interference intensity;

[0047] Specifically, the background perturbation confidence value (C tThis reflects the intensity of background changes and serves as one of the bases for the signal fusion module to judge the interference intensity. The calculation method of this background disturbance confidence value depends on the output of optical flow estimation and background modeling. By quantifying the disturbance intensity, it can provide an objective and accurate basis for subsequent interactive judgment.

[0048] If the background disturbance intensity is high, i.e., C t If the value is high, the system should suppress the credibility of the user input signal to prevent the background change from being mistakenly judged as a valid user behavior, thereby triggering an unnecessary system response. This process is achieved by dynamically adjusting the sampling weight of the modal signal, so that the system responds more cautiously to interference signals when the background disturbance is strong.

[0049] For example, in highly dynamic scenarios, such as when a dome screen video displays an explosion effect, the drastic changes in the background can easily cause user actions (such as raising a hand or turning a head) to be misinterpreted as system interaction commands. The background motion isolation module can identify background changes in real time to ensure that user actions are not misinterpreted in such highly turbulent scenarios.

[0050] Low-interference interactive environment: If the background changes are relatively stable, such as when the user is watching a static or slowly changing dome scene, the system can increase its sensitivity to user input signals to ensure smooth interactive response. Optionally, when the user controls the player through voice commands, the background motion isolation mechanism will ensure that the user's voice signal is not interfered with by subtle changes in the background.

[0051] For the confidence value of background disturbance (C) t In other words, this enables the system to exhibit adaptability under different background disturbances, for example, when C t When the value is relatively high (approaching 1), the system can choose to temporarily not respond to user input to avoid problems caused by accidental triggering; while in C... t When the input is low (approaching 0), the system can respond to user input normally, ensuring smooth interaction.

[0052] Furthermore, the working process of the interactive window period dynamic adjustment module is as follows:

[0053] The temporal characteristics of action and speech input signals are collected over multiple sampling periods. By comparing them with the action probability threshold α and the speech energy threshold β, it is determined whether the conditions for entering the interaction window are met.

[0054] If the conditions are met, the system will remain in an interactive response state until a signal indicating the end of the behavior is detected.

[0055] Among them, the interaction window period (W) tThe window period refers to the time period during which the system responds to user input within a specific time frame. In a dome screen environment, due to the potential for drastic visual changes in the projected content, the system needs to dynamically adjust this window period to adapt to user behavior. This dynamic adjustment mechanism ensures that the system can more sensitively capture user input signals and determine whether they belong to the valid interaction period based on the timing characteristics of the input. If the user sends a signal outside the interaction period, the system should avoid triggering a false response.

[0056] During multiple sampling periods, the system simultaneously collects user gesture signals and voice input signals. Within each sampling period, the system analyzes the temporal characteristics of these input signals and compares them with preset action probability thresholds α and voice energy thresholds β. These two thresholds are used to determine whether the user is performing a valid interactive action and voice input, respectively. If the user's input meets these two threshold requirements, the system considers the current time to be within a valid interaction window and responds accordingly.

[0057] In dome-screen interactive scenarios, users may simultaneously perform physical gestures and voice input, and the timing of these inputs is crucial for determining the interaction window. The system needs to analyze the temporal coordination between action and voice signals. For example, if a user issues a voice command to "fast forward" and simultaneously makes a gesture command, the system needs to determine whether these two actions fall within the same valid interaction window. Based on this, the system dynamically adjusts the response time of the interaction window to ensure that the input signals are processed correctly.

[0058] After recognizing a valid interaction signal, the system enters an interactive response state and maintains this state until the system detects a signal indicating the end of the behavior. Specifically, the dynamic adjustment function of the interaction window will extend or shorten accordingly based on the duration of the user's behavior to ensure that the system provides timely feedback to all valid signals. For example, if the user's physical movement is interrupted at a certain moment, the system can dynamically shorten the interaction window based on this change to avoid triggering incorrect responses when there is invalid input.

[0059] For example, a user is watching a movie and controls the playback progress using the voice command "fast forward." If the system simultaneously receives a vague gesture from the user (e.g., the user waves their hand forward, but the movement is unclear), the interaction window period dynamic adjustment module can analyze the timing characteristics to determine whether the gesture is a valid input, thereby ensuring that the system does not incorrectly respond to the unclear gesture.

[0060] To achieve temporal input comparison and effective window period determination, the system can introduce a convolutional neural network (CNN) to extract the spatiotemporal features of the input signal. CNN can effectively capture the temporal patterns in the user input signal through hierarchical convolution operations, helping the system to determine whether an effective interaction period has been entered.

[0061] The system will match the collected user behavior data with a preset time-series feature library to determine whether the current signal belongs to a valid interaction period. This process can effectively avoid false triggering of responses and ensure that the system responds to user input in a timely manner.

[0062] The system dynamically determines whether user input meets the standards for the interaction period by setting an action probability threshold α and a voice energy threshold β. Only when both thresholds are met will the system consider the user's input signal to be in a valid interaction period and respond accordingly.

[0063] Specifically, the action determination threshold α is defined as the action probability value output by the action classification model (such as 3DCNN or ST-GCN), and the recommended setting range is α∈[0.65,0.85].

[0064] Speech energy threshold β: defined as the short-time energy RMS value of the sound signal, with a suggested range of β∈[0.02,0.05] (the unit can be the relative energy normalized value).

[0065] Furthermore, the modal source tracking module operates as follows:

[0066] By performing feature vector correlation modeling on modal signals from different times and sources, and outputting a signal source credibility score based on attention weights;

[0067] This score is used as a reference when the signal fusion module performs weighted calculations to optimize the final response output.

[0068] Specifically, firstly, the modal source tracking module extracts features from each modal signal, and the system then correlates and models these feature vectors.

[0069] Secondly, each input signal generates an attention weight to evaluate its contribution to the current task. By calculating these weights, the system can determine which signals come from the user's real interaction behavior and which signals may be background interference or misidentification by the system. In order to more accurately identify the source of the signal and evaluate its credibility, the system adopts the multi-head attention mechanism in the Transformer network. This mechanism can highlight the importance of each signal source in the current interaction by weighting different signal sources.

[0070] Furthermore, if the signal has timing deviations or inconsistencies, the system will reduce the reliability of the signal and perform corresponding corrections.

[0071] Finally, after the above process is completed, the modal source tracking module will output a signal source credibility score based on the timing comparison and attention weighting of each modal signal. This score reflects the credibility of each modal signal source. By weighting the credibility scores of different signal sources, the system can determine which input signals are reliable and which may be caused by background interference or false triggering.

[0072] The output of the modal source tracking module (i.e., the confidence score of the signal source) will directly affect the decision-making process of other functional modules of the system, especially the signal fusion module. The signal fusion module will use the confidence score from the modal source tracking module as one of the important parameters to evaluate the final response of the multimodal input signal.

[0073] The confidence score of a signal source also affects whether the system issues a corresponding response command. For example, if the confidence score of a certain modal signal is very low, the system may choose to ignore the signal to avoid executing a falsely triggered command.

[0074] Furthermore, the working process of the dome screen change trend prediction module is as follows:

[0075] Features from the past n frames of images are extracted using a sliding window approach, and the extracted features are jointly processed by a convolutional neural network and a long short-term memory network.

[0076] The convolutional neural network (CNN) is used to extract spatial features from a single frame image;

[0077] The Long Short-Term Memory (LSTM) network is used to model the temporal variation patterns of image sequences;

[0078] Finally, the predicted output is mapped to a future disturbance probability value, which is used to determine whether the system enters a disturbance shielding state.

[0079] Specifically, features from the past n frames of images are first extracted using a sliding window approach;

[0080] Secondly, each frame of the image is processed by a CNN to extract spatial features. The CNN can learn and extract key visual features from the image, such as edges, textures and shapes. These features are crucial for capturing the visual changes of the dome screen.

[0081] Meanwhile, the extracted image features are then used for temporal modeling through an LSTM network. The LSTM network can effectively process sequential data and capture temporal patterns. Especially when dealing with background changes over a long period of time, the memory function of LSTM enables the system to predict future perturbation trends based on past changes. For example, when the background changes rapidly, LSTM can learn this pattern from historical temporal features, thereby improving the accuracy of predicting future changes.

[0082] Furthermore, the output of the LSTM, after further processing, is mapped to a perturbation probability value P. t This probability value P t This indicates the probability that the background will change drastically in the future. If the disturbance probability value exceeds the set threshold (e.g., 0.65), the system will activate the interference shielding mechanism to avoid false triggering caused by background changes. For example, when the dome screen displays an explosion scene or a strong flashing effect, the system will automatically judge that the background change is too drastic and take measures to shield the interactive input signal to prevent unnecessary misoperation. When the disturbance probability value is lower than the set threshold, the system will restore or maintain the response state.

[0083] Finally, the dome screen change trend prediction module determines whether to enter interference shielding mode based on the disturbance probability value. If the background disturbance intensity is large, the system will shield all or part of the interactive signal input to ensure that the user's commands will not be misunderstood or incorrectly responded to due to background changes. Conversely, when the background is stable or the disturbance is small, the system continues to maintain normal interactive mode.

[0084] For example, when watching dome screen content with explosion effects, strong background changes can quickly cause interference to modal signals. The dome screen change trend prediction module can identify this background change trend and activate the interference shielding mechanism in time to avoid misjudging the user's actions or voice input as interactive signals of the system.

[0085] Alternatively, transitions or flash effects often cause abrupt changes in the background. The system may mistakenly identify this change as a user input signal, leading to modal drift or false triggering of commands. By predicting these changes in advance, the system can mask invalid input before the transition begins, ensuring smooth interaction.

[0086] Furthermore, the operation process of the modal anti-interference filtering module is as follows:

[0087] The collected user input modal signals are input to a filter implemented by a neural network, and an interference sample set generated by a diffusion generation model is introduced.

[0088] The system trains the parameters of the filtering function by minimizing the weighted sum of the loss of normal samples and the loss of interfering samples.

[0089] The working principle of the modal anti-interference filtering module mainly consists of two parts: generating anti-interference samples and optimizing the filter function. The specific process is as follows:

[0090] Input Signal and Interference Sample Generation: The system first receives signals from different modal input channels, including but not limited to user voice input, body movement input, and gaze trajectories. However, under the influence of complex backgrounds, these signals may be interfered with by changes in the visual background, leading to false triggers. Therefore, to improve the system's ability to identify interference, a diffusion adversarial generative model is introduced. The goal of this model is to generate challenging interference samples and, by simulating real interference signals, train the system to respond accurately in inconsistent environments.

[0091] Generative Adversarial Training: The diffusion adversarial generative model is built on the principles of Generative Adversarial Networks (GANs). This model consists of two parts: a generator and a discriminator. The generator is responsible for creating highly disruptive adversarial examples that mimic various interference signals that might appear in a dome-shaped environment. The discriminator is responsible for determining whether the input signal is normal or disruptive and feeding the result back to the generator. In this way, the generator continuously optimizes to generate more challenging interference signals, thereby improving the robustness of the filter.

[0092] Neural Network Filter Training: By inputting the generated interfering samples and normal samples into a filter implemented by a neural network, the system begins training the filter. During training, the system attempts to minimize the weighted sum of the loss functions of the normal and interfering samples. Specifically, the system adjusts the parameters of the filter function based on the differences between the adversarial samples and normal samples, enabling the filter to effectively suppress interference and extract user behavior signals when encountering complex background disturbances.

[0093] Furthermore, the operation process of the signal fusion module is as follows:

[0094] Using the interaction state as the gate variable, the background disturbance intensity, modal signal source credibility, signal-to-noise ratio of the filter function, and disturbance prediction probability are comprehensively evaluated.

[0095] If the above parameters indicate that the background is stable, the signal is reliable, and the signal-to-noise ratio is good, the module will perform weighted summation on each modal signal to generate a fused expression value, and determine whether to issue a response command based on this value.

[0096] The core of the signal fusion module is to perform reliable quantification and weighted fusion judgment on the final system response decision under multimodal mixed input and complex background disturbance environment. Its role is to integrate the output of each functional module and finally determine whether the system executes the user interaction command.

[0097] Let the modal fusion output at the current moment be The fusion function is designed as follows:

[0098] ;

[0099] in, As a fusion response strength indicator, t is the time index of the current frame. The system decides whether to execute the user command based on whether this value exceeds the threshold δ.

[0100] W t W is an interaction window gating variable with a Boolean value ∈ {0,1}, indicating whether the user is currently in an interaction window. t When the value is 0, all inputs will be suppressed, and inputs outside the interaction window will be completely ignored;

[0101] C t C represents the confidence value for background perturbation. t ∈[0,1], the closer to 1, the more drastic the background changes and the greater the interference. The stronger the background perturbation, the higher C becomes. t →1, then (1-C t When the input term approaches 0, its weight in the total response is reduced.

[0102] S t i To assess the reliability of modal signal sources, a Transformer network can be used to score the signal source for each mode, S. t i ∈[0,1], evaluated by factors such as signal quality and source localization error, the weighting coefficient increases when the source is reliable. Preferably, when S t i A value less than 0.4 indicates a high probability of interference; when 0.4 ≤ S t i A value ≤0.7 indicates moderate reliability; when S... t i A value greater than 0.7 indicates a highly reliable signal;

[0103] f θ (M t i The output value of the modal filter represents the strength of the "effective signal" obtained after modality i (representing the i-th modal channel, such as speech, gesture, head movement, etc.) is processed by the anti-interference filter with training parameters θ, reflecting noise immunity and stability. θ (M t i The input to the function is the original modal signal (M). t i The output is the modal representation with the signal-to-noise ratio maximized;

[0104] In order to achieve effective processing of the input signal, f θ (M t i The following network architecture is typically used:

[0105] Input layer: Input signal (M) t i After preprocessing, the data is fed into the network. This input is typically a representation of the signal source in the time or frequency domain, which can be sampled time-series data or a feature-extracted representation of the signal.

[0106] Feature Extraction Layer: This layer uses a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN) to extract features from the signal. CNNs are suitable for spatial feature extraction of image or video signals, while RNNs are suitable for signals with strong temporal sequence (such as speech signals). The goal of this layer is to extract key information from the original modal signal;

[0107] Adversarial Training Layer: By using Generative Adversarial Networks (GANs) or other similar adversarial training methods, the system can simulate interference signals in real-world scenarios and train the network to identify and remove these interference signals. This layer generates perturbation signals and performs adversarial training against the original signals, enabling the network to adapt to and resist background noise and environmental interference.

[0108] Filter layer: After training, the network's filter layer can process the input signal, suppress unnecessary noise or disturbances, output an optimized signal, and enhance the signal-to-noise ratio.

[0109] Output layer: Generates the processed signal (M) t i This refers to the filtered signal output. These signals will be used in the subsequent weighted fusion module to determine whether to execute the interactive command.

[0110] P t To predict the probability of future disturbances, the dome screen system predicts the likelihood of an impending dramatic change, P. t ∈[0,1], the closer to 1, the stronger the perturbation in future frames, and the current instruction should be suppressed.

[0111] In a scenario where the dome screen flashes white rapidly and the user makes large hand gestures, due to the intense background disturbance, C t The highest value is 0.85, therefore, W t =1 (in the interaction period), and since there is no voice input, therefore, S t i The gesture modality score is high, at 0.9, and the modality filter output value f θ (M t i The value is 0.95, and the prediction of dome-shaped dramatic changes leads to P. t The highest value is 0.8, and ultimately, because P t With C t At the same time, it is relatively high. The value was lowered, and the final calculated result was approximately 0.0256. This value was significantly lower than the threshold δ, so the system refused to respond to this mode, thus avoiding visual misjudgment due to the explosion and helping to avoid false triggering.

[0112] The design goal of this signal fusion module is to maximize response accuracy and reduce the risk of false triggering in scenarios where complex background changes and real user interactions coexist.

[0113] Furthermore, the system executes the corresponding modal response command only when the fused expression value exceeds a system-set threshold;

[0114] If the system's set threshold is not reached, the system will remain in a non-interactive, silent state.

[0115] That is when When the input mode is ≥δ, the system will only execute a response if the current input mode is considered reliable, the background is stable, and there is no interference.

[0116] when When the value is less than δ, the current input is considered unreliable, and the system remains silent.

[0117] Generally speaking, the higher the threshold δ, the lower the false trigger rate of the system, but the system may miss the real user intent, reduce the interaction sensitivity, and lead to missed recognition. The lower the threshold δ, the higher the interaction sensitivity, but the anti-interference ability decreases. The system may still trigger the response when the modal signal is not reliable or the disturbance is severe, which can easily cause false responses. Therefore, preferably, δ∈[0.45,0.75].

[0118] More specifically, when the user first enters the scene (without prior data), δ is set to 0.6. If the system judges that the current scene is relatively stable, it is gradually reduced to 0.5 to enhance sensitivity. If the background disturbance is frequent, it is dynamically increased to 0.7 to improve defense.

[0119] Furthermore, the system also includes a modal perception module, which comprises an array of motion sensors aligned by a time synchronization mechanism, a directional voice receiving array, and a gaze trajectory tracking unit, for synchronously acquiring the user's motion input signals, voice input signals, and visual attention information.

[0120] Motion sensor array: This array is responsible for capturing the user's body movements, such as gestures and body postures. Through high-frequency sensor data acquisition, the system can accurately determine the user's motion characteristics and convert them into input signals.

[0121] Directional voice receiver array: Used to acquire the user's voice signal and filter noise through a directional microphone array to ensure the accuracy of voice recognition. In a dome screen environment, voice input may overlap with background sound effects, so directional reception can improve the signal-to-noise ratio of voice input.

[0122] Gaze trajectory tracking unit: This unit records the user's visual attention points through eye-tracking technology. The gaze trajectory can serve as an auxiliary signal to help the system determine whether the user is focusing their attention on a specific area or content, thereby affecting the modal response.

[0123] These modules align various input signals through a time synchronization mechanism. Time synchronization ensures that information from different sensors can be accurately compared on the same time axis, thereby reducing misjudgments caused by timing misalignment. Through this synchronization, the system can capture all the details of user interaction and respond in a timely manner.

[0124] like Figure 2 As shown, the present invention also provides an interactive dome screen viewing method, comprising the following steps:

[0125] The system collects user motion input signals, voice input signals, and visual attention information, and uses the motion sensor array, directional voice receiving array, and gaze trajectory tracking unit in the modal perception module to achieve synchronous acquisition.

[0126] Optical flow estimation and background modeling are performed on the dome video stream through the background motion isolation module, the spatiotemporal variation characteristics of the current image are extracted and the confidence of background perturbation is calculated;

[0127] Based on the user's body movements and voice input signals, the interaction window dynamic adjustment module dynamically adjusts the response time window of the interaction window according to the temporal characteristics of the body movement and voice input signals.

[0128] The modal source tracking module is used to perform feature vector correlation modeling for input signals of different modes and dynamically evaluate the credibility of each modal signal source, providing a basis for signal fusion;

[0129] The dome screen change trend prediction module integrates convolutional neural networks and long short-term memory networks to predict the future perturbation trend of the dome screen image background and outputs the future perturbation probability value.

[0130] An anti-interference sample set is generated by a modal anti-interference filtering module, and the robustness parameters of the filter are optimized by a neural network to filter out interference signals.

[0131] The processing results of each module are integrated through the signal fusion module to comprehensively evaluate the background disturbance intensity, modal signal source credibility, interactive window state, signal-to-noise ratio and disturbance prediction probability, and generate fused expression values.

[0132] In the description of this invention, it should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions provided in this disclosure can be achieved, and no limitation is imposed herein.

[0133] The above description is merely a preferred embodiment of the present invention and does not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An interactive dome theater viewing system based on AI multimodal data processing, characterized in that, include: The background motion isolation module analyzes and processes the dome video stream signal through optical flow estimation, extracts the spatiotemporal change characteristics of the current frame image, and calculates the background perturbation confidence level in conjunction with the background modeling process. The interactive window period dynamic adjustment module is used to dynamically adjust the system's response state time window based on the temporal coordination between the user's body movement input signal and voice input signal. The modal source tracking module is based on the Transformer neural network architecture. It performs feature vector correlation modeling on modal signals from different input channels through a multi-head attention mechanism and dynamically evaluates the credibility of each modal signal source. The dome screen change trend prediction module integrates a convolutional neural network (CNN) and a long short-term memory network (LSTM) to predict the future perturbation trend of the dome screen image background. A modal anti-interference filtering module generates anti-interference samples based on a diffusion adversarial generative model training method, and optimizes the robustness parameters of the filter function through a neural network; The signal fusion module integrates the processing results from each module and comprehensively evaluates the background disturbance intensity, the reliability of the modal signal source, and the interactive window status.

2. The interactive dome theater viewing system based on AI multimodal data processing according to claim 1, characterized in that: The working process of the background motion isolation module is as follows: The optical flow estimation process is used to extract motion vector information of each pixel within the current image frame; The background modeling process calculates the motion vector difference based on the expected position of the predicted pixel and obtains the perturbation intensity, and then further generates the background perturbation confidence value; The confidence level value serves as one of the bases for the signal fusion module to determine the interference intensity.

3. The interactive dome theater viewing system based on AI multimodal data processing according to claim 1, characterized in that: The working process of the interactive window period dynamic adjustment module is as follows: The temporal characteristics of action and speech input signals are collected over multiple sampling periods. By comparing them with the action probability threshold α and the speech energy threshold β, it is determined whether the conditions for entering the interaction window are met. If the conditions are met, the system will remain in an interactive response state until a signal indicating the end of the behavior is detected.

4. The interactive dome theater viewing system based on AI multimodal data processing according to claim 1, characterized in that: The modal source tracking module operates as follows: By performing feature vector correlation modeling on modal signals from different times and sources, and outputting a signal source credibility score based on attention weights; This score is used as a reference when the signal fusion module performs weighted calculations to optimize the final response output.

5. The interactive dome theater viewing system based on AI multimodal data processing according to claim 1, characterized in that: The working process of the dome screen change trend prediction module is as follows: Features from the past n frames of images are extracted using a sliding window approach, and the extracted features are jointly processed by a convolutional neural network and a long short-term memory network. The convolutional neural network is used to extract spatial features from a single frame image; The long short-term memory network is used to model the temporal variation patterns of image sequences; Finally, the predicted output is mapped to a future disturbance probability value, which is used to determine whether the system enters a disturbance shielding state.

6. The interactive dome theater viewing system based on AI multimodal data processing according to claim 1, characterized in that: The operation process of the modal anti-interference filtering module is as follows: The collected user input modal signals are input to a filter implemented by a neural network, and an interference sample set generated by a diffusion generation model is introduced. The system trains the parameters of the filtering function by minimizing the weighted sum of the loss of normal samples and the loss of interfering samples.

7. An interactive dome theater viewing system based on AI multimodal data processing according to claim 6, characterized in that: The operation process of the signal fusion module is as follows: Using the interaction state as the gate variable, the background disturbance intensity, modal signal source credibility, signal-to-noise ratio of the filter function, and disturbance prediction probability are comprehensively evaluated. If the above parameters indicate that the background is stable, the signal is reliable, and the signal-to-noise ratio is good, the module will perform weighted summation on each modal signal to generate a fused expression value, and determine whether to issue a response command based on this value.

8. The interactive dome theater viewing system based on AI multimodal data processing according to claim 7, characterized in that: The system executes the corresponding modal response command only when the fusion expression value exceeds the system's set threshold; If the system's set threshold is not reached, the system will remain in a non-interactive, silent state.

9. The interactive dome theater viewing system based on AI multimodal data processing according to claim 1, characterized in that: The system also includes a modal perception module, which comprises a motion sensor array, a directional voice receiving array, and a gaze trajectory tracking unit aligned by a time synchronization mechanism, for synchronously acquiring the user's motion input signals, voice input signals, and visual attention information.

10. An interactive dome theater viewing method, used in an interactive dome theater viewing system based on AI multimodal data processing as described in any one of claims 1-9, characterized in that, Includes the following steps: The system collects user motion input signals, voice input signals, and visual attention information, and uses the motion sensor array, directional voice receiving array, and gaze trajectory tracking unit in the modal perception module to achieve synchronous acquisition. Optical flow estimation and background modeling are performed on the dome video stream through the background motion isolation module, the spatiotemporal variation characteristics of the current image are extracted and the confidence of background perturbation is calculated; Based on the user's body movements and voice input signals, the interaction window dynamic adjustment module dynamically adjusts the response time window of the interaction window according to the temporal characteristics of the body movement and voice input signals. The modal source tracking module is used to perform feature vector correlation modeling for input signals of different modes and dynamically evaluate the credibility of each modal signal source, providing a basis for signal fusion; The dome screen change trend prediction module integrates convolutional neural networks and long short-term memory networks to predict the future perturbation trend of the dome screen image background and outputs the future perturbation probability value. An anti-interference sample set is generated by a modal anti-interference filtering module, and the robustness parameters of the filter are optimized by a neural network to filter out interference signals. The processing results of each module are integrated through the signal fusion module to comprehensively evaluate the background disturbance intensity, modal signal source credibility, interactive window state, signal-to-noise ratio and disturbance prediction probability, and generate fused expression values.

Images