Multimodal fusion immersive theater interaction control system
Through the immersive theater interactive control system with multimodal fusion, the multimodal input system and deep learning model are used to process audience data simultaneously and generate personalized control instructions, solving the delay problem caused by multimodal data acquisition in immersive theaters, and improving the audience's immersive experience and feedback speed.
Patent Information
- Application Number
- CN202510583353.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2025-08-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The delay problem of feedback delay in the sound and light environment caused by the complex multimodal data acquisition in an immersive theater affects the audience's immersive experience effect.
The multi-modal input system is used to collect audience data, and the modal data is synchronized and integrated through the multi-modal fusion engine of the data processing layer. Combined with the deep learning model, the decision-making layer builds the audience portrait and generates control instructions, the execution layer adjusts stage feedback, and uses a simple instruction database to optimize instruction processing to reduce delay.
It improves the audience's immersive experience, reduces the delay in the theater's interaction, enhances feedback speed and personalized control, and improves the service experience of the immersive theater.
Smart Images
Figure CN120491817A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of immersive theatre interaction systems, and more particularly to a multimodal fusion immersive theatre interaction control system. Background Art
[0002] Immersive theatre is a form of theatre that transcends traditional boundaries between audience and performance. Through multi-sensory interaction, exploration of physical space, and audience participation in the narrative, it transforms the audience from a mere spectator into a character within the story, providing a highly personalized experience. Its core concept is to "break the fourth wall," the invisible barrier between performer and audience.
[0003] The sound, light, and environment in an immersive theater directly affect the audience's immersive experience, and the interactive control system can easily control the feedback of sound, light, and environment. However, directly analyzing multimodal data layer by layer is likely to affect the delay of sound, light, and environment feedback as the number of audience members increases and the complexity of collecting modal data, greatly reducing the audience's immersive experience. Summary of the Invention
[0004] In view of the problems existing in the prior art, the purpose of the present invention is to provide an immersive theater interactive control system with multimodal fusion to solve the background technical problems.
[0005] To achieve the above objectives, the present invention adopts the following technical solutions:
[0006] A multi-modal fusion immersive theater interactive control system, comprising:
[0007] A monitoring layer, comprising a multimodal input system for collecting various modal data of audience members seated in the theater and uploading the collected modal data to a data processing layer;
[0008] The data processing layer includes a data processing module and a multimodal fusion engine. The data processing module is used to process the modal data uploaded by the multimodal input system in real time. The multimodal fusion engine is used to synchronize the spatiotemporal systems of each modal data and integrate the multimodal signals using a deep learning model.
[0009] The decision-making layer includes an audience portrait module, a theater dynamic adjustment module, and an instruction selection module. The audience portrait module is used to construct and update the dynamic feature model of individual / group audiences in real time based on the modal signals fused by the multimodal fusion engine; the theater dynamic adjustment module adjusts the theater stage according to the dynamic feature model generated by the audience portrait module and generates control instructions; the instruction screening module removes duplicate and conflicting instructions from the multiple control instructions generated by the theater dynamic adjustment module;
[0010] The execution layer is used to receive the control instructions filtered by the instruction selection module and control and adjust the stage's feedback to the audience according to the instructions.
[0011] As a further description of the above technical solution:
[0012] It also includes a simple instruction database and an instruction selection module. The simple instruction database is used to store simple instructions, and each simple instruction corresponds to one or more integrated multimodal signal data; the instruction selection module is used to select the corresponding simple instruction according to the integrated multimodal signal data generated by the data processing layer, and directly send it to the execution layer to execute the instruction.
[0013] As a further description of the above technical solution:
[0014] The simple instruction database is connected to the decision layer signal, and is used to receive the instructions generated by the decision layer for integrating the multimodal signal data, and update the stored data in the simple instruction database.
[0015] As a further description of the above technical solution:
[0016] The multimodal input system includes a depth camera, a microphone, and an environmental sensor, and is used to obtain various modal data of the audience through vision, hearing, and touch.
[0017] As a further description of the above technical solution:
[0018] The screening principle of the instruction screening module is to delete duplicate instructions and retain one of the redundant instructions, and select the instruction with the highest number of votes according to the voting mechanism of the minority obeys the majority principle for conflicting instructions. When the instruction with the highest number of votes is not unique, one of the instructions is randomly selected for execution.
[0019] As a further description of the above technical solution:
[0020] The instruction screening module further includes a role access unit, which is used to access theater plot interpretation instructions and personnel control instructions. The priority of the plot interpretation instructions and personnel control instructions is higher than the control instructions generated by the theater dynamic adjustment module.
[0021] As a further description of the above technical solution:
[0022] The execution layer includes stage lighting, props, audio, video, seats, ambient temperature control and fans.
[0023] As a further description of the above technical solution:
[0024] The simple instruction database includes an input unit, which presets control instructions for corresponding scenarios according to the audience's post-viewing experience and stores them in the simple instruction database.
[0025] Compared with the prior art, the advantages of the present invention are:
[0026] (1) This solution can adjust the state of the theater stage according to the individual and group intentions of the audience during viewing, thereby improving the audience's viewing experience, increasing the audience's viewing immersion, and increasing the service experience effect of the immersive theater.
[0027] (2) This solution greatly reduces the delay of theater interaction, improves the feedback speed of the theater execution layer, and enhances the audience's immersive experience. It manually inputs corresponding instructions based on the audience's viewing experience and questionnaires, which is more humane and personalized. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 Schematic diagram of the interactive control principle of the present invention;
[0029] Figure 2 Schematic diagram of the interactive control process of the present invention.
[0030] Description of the numbers in the figure:
[0031] 1. Monitoring layer; 11. Multimodal input system; 2. Data processing layer; 21. Data processing module; 22. Multimodal fusion engine; 3. Decision layer; 31. Audience portrait module; 32. Theater dynamic adjustment module; 33. Command screening module; 4. Execution layer; 5. Simple command database; 51. Entry unit; 6. Command selection module. DETAILED DESCRIPTION
[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention;
[0033] See also Figure 1 and Figure 2 , the present invention provides embodiment 1:
[0034] A multimodal fusion immersive theater interactive control system includes: a monitoring layer 1, which includes a multimodal input system 11. The multimodal input system 11 is used to collect various modal data of the audience in the seats in the theater and upload the collected modal data to the data processing layer 2; the multimodal input system 11 includes a depth camera, a microphone and an environmental sensor, which is used to obtain various modal data of the audience through vision, hearing and touch.
[0035] Data processing layer 2, data processing layer 2 includes a data processing module 21 and a multimodal fusion engine 22. The data processing module 21 is used to process the modal data uploaded by the multimodal input system 11 in real time; the multimodal fusion engine 22 is used to synchronously align the spatiotemporal systems of each modal data and use a deep learning model to integrate multimodal signals.
[0036] Decision layer 3, decision layer 3 includes an audience portrait module 31, a theater dynamic adjustment module 32 and an instruction screening module 33. The audience portrait module 31 is used to construct and update the dynamic feature model of the individual / group audience in real time according to the modal signal fused by the multimodal fusion engine 22; the theater dynamic adjustment module 32 adjusts the theater stage according to the dynamic feature model generated by the audience portrait module 31 and generates control instructions; the instruction screening module 33 removes duplicate instructions and conflicting instructions from the multiple control instructions generated by the theater dynamic adjustment module 32.
[0037] The execution layer 4 is used to receive the control instructions filtered by the instruction filtering module 33 and control and adjust the stage's feedback to the audience according to the instructions; the execution layer 4 includes stage lighting, props, audio, video, seats, ambient temperature control and fans.
[0038] The multimodal input system 11 is used to collect modal data on the movements and sounds of theater audiences. The data processing layer 2 organizes and integrates the modal data. Based on this, the audience portrait module 31 generates a dynamic feature model of the audience and analyzes and determines the audience's intentions. For example, the "excited" label of audience A = visual smile + auditory accelerated speech + tactile increased heart rate.
[0039] The theater dynamic adjustment module 32 determines the adjustment plan according to the audience's intention, and the instruction screening module 33 selects instructions to execute, so as to avoid the execution layer 4 from executing multiple times or failing to execute due to conflicting instructions.
[0040] It can adjust the state of the theater stage according to the individual and group viewing intentions of the audience, thereby improving the audience's viewing experience, increasing the audience's viewing immersion, and increasing the service experience effect of the immersive theater.
[0041] The screening principle of the instruction screening module 33 is to delete duplicate instructions and retain one of the redundant instructions, and select the instruction with the highest number of votes according to the voting mechanism of the minority obeys the majority principle for conflicting instructions. When the instruction with the highest number of votes is not unique, one of the instructions is randomly selected for execution.
[0042] The instruction screening module 33 further includes a role access unit 331 , which is used to access theater plot interpretation instructions and personnel control instructions. The priority of the plot interpretation instructions and personnel control instructions is higher than the control instructions generated by the theater dynamic adjustment module 32 .
[0043] Due to the need to advance the script in the theater, specific instructions need to be executed at key nodes to advance the plot while ensuring the integrity and smoothness of the plot. The specific instructions are the theater plot interpretation instructions. At this time, the stage is controlled based on the priority execution of the theater plot interpretation instructions, and the personnel control instructions are instructions issued by backstage personnel for maintenance and adjustment, which facilitates backstage personnel to control the stage and ensure the interpretation experience and safety.
[0044] See also Figure 1 , the present invention further provides Example 2 based on Example 1:
[0045] The simple instruction database 5 and the instruction selection module 6, the simple instruction database 5 is used to store simple instructions, each simple instruction corresponds to one or more integrated multimodal signal data; the instruction selection module 6 is used to select the corresponding simple instruction according to the integrated multimodal signal data generated by the data processing layer 2, and directly send it to the execution layer 4 to execute the instruction.
[0046] The integrated multimodal signal transmitted from the data processing layer 2 to the decision layer 3 is synchronously obtained through the instruction selection module 6, and the integrated multimodal signal data with an overlap degree of more than 90% is selected in the simple instruction database 5 according to the integrated multimodal signal. If there is highly overlapping data, the execution instruction of the corresponding data is selected and sent directly to the execution layer 4 through the instruction selection module 6 for execution. The decision layer 3 stops the decision operation of the integrated multimodal signal data and executes the decision analysis of the next multimodal signal data, thereby greatly reducing the theater interaction delay, improving the feedback speed of the theater execution layer 4, and improving the audience's immersive experience.
[0047] The simple instruction database 5 is connected to the decision layer 3 by signal, and is used to receive the instructions generated by the decision layer 3 for integrating the multimodal signal data, and update the stored data in the simple instruction database 5;
[0048] The analysis and decision-making of the decision-making layer 3 on various integrated multimodal signal data are saved in the simple instruction database 5 to try to update and enrich the execution instructions in the simple instruction database 5, thereby improving the timeliness of feedback of the interactive control system.
[0049] The simple instruction database 5 includes an input unit 51, which presets control instructions for corresponding scenarios based on the audience's viewing experience and stores them in the simple instruction database 5;
[0050] According to the audience's viewing experience and questionnaire surveys, corresponding instructions are manually input, which is more humane and personalized.
[0051] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any person skilled in the art who, within the technical scope disclosed by the present invention, makes equivalent substitutions or modifications based on the technical solutions and improved concepts of the present invention shall be covered by the scope of protection of the present invention.
Claims
1. A multi-modal fusion immersive theater interactive control system, characterized by: include: A monitoring layer (1), wherein the monitoring layer (1) includes a multimodal input system (11), wherein the multimodal input system (11) is used to collect various modal data of audience members sitting in seats in the theater and upload the collected modal data to the data processing layer (2); A data processing layer (2), the data processing layer (2) comprising a data processing module (21) and a multimodal fusion engine (22), the data processing module (21) being used for real-time processing of modal data uploaded by the multimodal input system (11); the multimodal fusion engine (22) being used for synchronously aligning the spatiotemporal systems of various modal data and integrating multimodal signals using a deep learning model; A decision layer (3), comprising an audience portrait module (31), a theater dynamic adjustment module (32) and an instruction screening module (33), wherein the audience portrait module (31) is used to construct and update a dynamic feature model of an individual / group audience in real time based on the modal signal fused by the multimodal fusion engine (22); the theater dynamic adjustment module (32) adjusts the theater stage based on the dynamic feature model generated by the audience portrait module (31) and generates control instructions; and the instruction screening module (33) removes duplicate instructions and conflicting instructions from the multiple control instructions generated by the theater dynamic adjustment module (32); The execution layer (4) is used to receive the control instructions filtered by the instruction filtering module (33) and control and adjust the stage's feedback to the audience according to the instructions.
2. The multimodal fusion immersive theater interactive control system according to claim 1, characterized in that: The system further comprises a simple instruction database (5) and an instruction selection module (6). The simple instruction database (5) is used to store simple instructions, each simple instruction corresponding to one or more integrated multimodal signal data; the instruction selection module (6) is used to select a corresponding simple instruction based on the integrated multimodal signal data generated by the data processing layer (2), and directly send the simple instruction to the execution layer (4) for execution.
3. The multimodal fusion immersive theater interactive control system according to claim 2, characterized in that: The simple instruction database (5) is connected to the decision layer (3) by signal, and is used to receive instructions generated by the decision layer (3) for integrating multimodal signal data, and to update the stored data in the simple instruction database (5).
4. The multimodal fusion immersive theater interactive control system according to claim 1, characterized in that: The multimodal input system (11) includes a depth camera, a microphone and an environmental sensor, and is used to obtain various modal data of the audience through vision, hearing and touch.
5. The multimodal fusion immersive theater interactive control system according to claim 1, characterized in that: The screening principle of the instruction screening module (33) is to delete duplicate instructions and retain one of the redundant instructions, and select the instruction with the highest number of votes according to the voting mechanism of the minority obeys the majority principle for conflicting instructions. When the instruction with the highest number of votes is not unique, one of the instructions is randomly selected for execution.
6. The multimodal fusion immersive theater interactive control system according to claim 1, characterized in that: The instruction screening module (33) further includes a role access unit (331), and the role access unit (331) is used to access theater plot interpretation instructions and personnel control instructions, and the priority of the plot interpretation instructions and personnel control instructions is higher than the control instructions generated by the theater dynamic adjustment module (32).
7. The multimodal fusion immersive theater interactive control system according to claim 1, characterized in that: The execution layer (4) includes stage lighting, props, audio, video, seats, environmental temperature control and fans.
8. The multimodal fusion immersive theater interactive control system according to claim 2, characterized in that: The simple instruction database (5) includes an input unit (51), which presets control instructions for corresponding scenarios based on the audience's post-viewing experience and stores the preset control instructions in the simple instruction database (5).