Immersive Kunqu experience method and system for CAVE system based on multi-modal interaction

Abstract

The invention provides a CAVE system immersive Kunqu experience method and system based on multi-modal interaction, and the method comprises the steps: adjusting CAVE projection space parameters based on a visual editing interface, and synchronously updating a motion capture mapping rule; a role holographic image is projected through laser holographic equipment, and meanwhile, a dynamic background is generated by using a DLP projector; the method comprises the following steps: capturing space action data of a user through an infrared camera, collecting gesture electromyographic signals of the user through an electromyographic sensor, analyzing text data of music sung by the user through voice recognition equipment, and monitoring physiological state data of the user through a heart rate sensor; generating a virtual scene special effect through a motion capture mapping rule, and driving light and shadow hue adjustment and environment parameter adjustment; and activating the branch plot through the AI decision tree algorithm, and calling the associated content of the cultural knowledge graph, so that the real-time interaction between the natural action of the user and the projection content can be realized, the physiological immersion of the user can be enhanced, the personalized experience of the user is supported, and the interactivity is improved.

Description

Technical Field

[0001] This invention relates to the field of CAVE virtual simulation technology, specifically to a method and system for immersive Kunqu Opera experience in a CAVE system based on multimodal interaction. Background Technology

[0002] Traditional CAVE systems rely on multi-channel projection to achieve spatial immersion, but content switching requires restarting the program or re-importing the configuration file, resulting in poor flexibility, limited interaction methods, and reliance on peripherals such as controllers. This makes it difficult to support real-time interaction between the user's natural movements and the projected content, and lacks physiological immersion.

[0003] Current digitization of Kunqu Opera mainly relies on video recording or static 3D modeling, which has weak interactivity. Audiences are in a passive viewing state and cannot participate in the plot development or trigger scene changes through actions, resulting in a lack of personalized experience. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide an immersive Kunqu Opera experience method and system based on multimodal interaction in the CAVE system, which aims to solve the technical problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution: An immersive Kunqu Opera experience method based on a CAVE system using multimodal interaction includes the following steps: Adjust CAVE projection space parameters based on the visual editing interface and update motion capture mapping rules simultaneously; The scene file is loaded based on the CAVE projection space, and the holographic image of the character is projected through a laser holographic device, while a dynamic background is generated using a DLP projector. The system captures user spatial motion data using an infrared camera, collects user gesture electromyography signals using an electromyography sensor, analyzes the text data of the lyrics sung by the user using a voice recognition device, and monitors the user's physiological state data using a heart rate sensor. Based on the spatial motion data and the electromyographic signals, virtual scene effects are generated through the motion capture mapping rules, and the lighting and color tone are adjusted and environmental parameters are modified based on the lyrics and the physiological state data. Based on the user's gaze duration and action sequence, branch plots are activated through an AI decision tree algorithm, and related content is invoked from a cultural knowledge graph.

[0006] According to one aspect of the above technical solution, the specific steps of adjusting the CAVE projection space parameters based on the visual editing interface and synchronously updating the motion capture mapping rules include: In the UnityUGUI editing interface, ScriptableObject is used to achieve two-way data binding between the physical screen model and the motion capture rigid body model; The number of projection surfaces and their spatial coordinates are defined using the physical screen model. The key point coordinates are defined using the motion capture rigid body model, and the motion capture mapping rules are configured.

[0007] According to one aspect of the above technical solution, the specific steps of generating virtual scene effects based on the spatial motion data and the electromyographic signals through the motion capture mapping rules, and driving the adjustment of light and shadow tones and environmental parameters based on the lyrics and the physiological state data, include: The central processing module generates virtual scene effects based on the spatial motion data and the intensity of the electromyographic signals, using the motion capture mapping rules preset in the GPU rendering cluster. When the wrist swing angle is >60° and lasts for 0.5 seconds, particle effects are triggered. When the finger opening and closing frequency is >2 times / second, the scene depth parameters are adjusted. The environmental control device uses the color palette of the DLP projector to adjust the scene lighting and shadows based on the lyrics and physiological state data. It adjusts the temperature within the range of 15-30℃ using the temperature control device and simulates airflow with a wind speed of 0-5m / s using the airflow generator. When it detects that the user's heart rate variability has decreased by 30% for 2 minutes, it triggers synchronous adjustment of temperature and humidity.

[0008] According to one aspect of the above technical solution, the specific steps of activating branch plots and calling cultural knowledge graph related content based on user gaze duration and action sequence include: When the infrared camera tracks the user's gaze at the virtual prop for more than 15 seconds, the associated side story is activated, and the cultural knowledge graph information is displayed through the laser holographic device. When a continuous squatting and forward pushing sequence is detected, the story jumps to the designated scene.

[0009] According to one aspect of the above technical solution, the cultural knowledge graph information includes: Entity layer: 3D models of Kunqu Opera characters and virtual props; Relationship layer: The ownership relationship between props and the script; Attribute layer: vocal style analysis data, Ming Dynasty clothing system text.

[0010] This invention also provides an immersive Kunqu Opera experience system based on multimodal interaction using the CAVE system, comprising: Editing module: Used to adjust CAVE projection space parameters based on a visual editing interface and to update motion capture mapping rules synchronously; Projection module: used to load scene files based on the CAVE projection space, project holographic images of characters through a laser holographic device, and generate dynamic backgrounds using a DLP projector; Interaction module: used to capture user spatial motion data through infrared camera, collect user gesture electromyography signals through electromyography sensor, parse user chanted lyrics text data through voice recognition device, and monitor user physiological state data through heart rate sensor; Central processing module: used to generate virtual scene effects based on the spatial motion data and the electromyographic signals through the motion capture mapping rules, and to drive the adjustment of light and shadow tones and the adjustment of environmental parameters based on the lyrics and the physiological state data; Narrative module: Based on the user's gaze duration and action sequence, it activates branching storylines using an AI decision tree algorithm and calls upon related content from a cultural knowledge graph.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention enhances the flexibility of the CAVE system by adjusting CAVE projection space parameters through a visual editing interface and simultaneously updating motion capture mapping rules. It supports real-time scene switching and allows for real-time reconstruction of the projection space while updating motion capture mapping rules. Based on the spatial motion data and electromyographic signals, virtual scene effects are generated using the motion capture mapping rules. Based on the lyrics and physiological state data, lighting and color tone adjustments and environmental parameter regulation are driven, enabling real-time interaction between the user's natural movements and the projected content, and enhancing the user's physiological immersion. Based on the user's gaze duration and action sequence, branching storylines are activated using an AI decision tree algorithm, and related content is accessed through a cultural knowledge graph, thus supporting personalized user experiences and improving the interactivity between the user and digital images. Attached Figure Description

[0012] Figure 1 This is a flowchart of an immersive Kunqu Opera experience method based on a CAVE system using multimodal interaction, according to the present invention.

[0013] Figure 2 This is a structural block diagram of an immersive Kunqu Opera experience system based on multimodal interaction, using the CAVE system, according to the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but includes other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 like Figure 1 As shown, this invention provides a method for immersive Kunqu Opera experience using a CAVE system based on multimodal interaction, including the following steps S1-S5: S1: Adjust CAVE projection space parameters based on the visual editing interface and update motion capture mapping rules simultaneously; S2: Load the scene file based on the CAVE projection space, project the holographic image of the character through the laser holographic device, and generate a dynamic background using a DLP projector; The laser holographic device and the DLP projector work in parallel, and both are connected to the central processing module through an HDMI synchronization interface to ensure pixel-level alignment between the holographic image and the projected background.

[0017] S3: Captures user spatial motion data through an infrared camera, collects user gesture electromyography signals through an electromyography sensor, parses the text data of the lyrics sung by the user through a voice recognition device, and monitors the user's physiological state data through a heart rate sensor. The infrared camera is deployed at the top of the CAVE projection space, covering a 360° field of view; the electromyography sensor is worn on the user's wrist; the voice module uses an iFLYTEK microphone array; the heart rate sensor is worn through a chest strap; all sensors communicate with the central processing module via a USB 3.0 hub, with a data latency of ≤20ms.

[0018] S4: Based on the spatial motion data and the electromyographic signals, generate virtual scene effects through the motion capture mapping rules, and drive the adjustment of light and shadow tones and environmental parameters based on the lyrics and the physiological state data; S5: Based on the user's gaze duration and action sequence, activate branch plots through AI decision tree algorithm and call cultural knowledge graph related content.

[0019] Understandably, this invention enhances the flexibility of the CAVE system by adjusting CAVE projection space parameters through a visual editing interface and simultaneously updating motion capture mapping rules. This allows for real-time scene switching and real-time reconstruction of the projection space while simultaneously updating motion capture mapping rules. Based on the spatial motion data and electromyographic signals, virtual scene effects are generated using the motion capture mapping rules. Based on the lyrics and physiological state data, adjustments to lighting and color tones and environmental parameters are driven, enabling real-time interaction between the user's natural movements and the projected content, and enhancing the user's physiological immersion. Furthermore, based on the user's gaze duration and action sequence, branching storylines are activated using an AI decision tree algorithm, and related content is accessed via a cultural knowledge graph, thus supporting personalized user experiences and improving the interactivity between the user and digital images.

[0020] Furthermore, the specific steps for adjusting the CAVE projection space parameters based on the visual editing interface and synchronously updating the motion capture mapping rules include: In the UnityUGUI editing interface, ScriptableObject is used to achieve two-way data binding between the physical screen model and the motion capture rigid body model; The number of projection surfaces and their spatial coordinates are defined using the physical screen model. The key point coordinates are defined using the motion capture rigid body model, and the motion capture mapping rules are configured.

[0021] Understandably, when the number of projection surfaces is modified in the UnityUGUI editing interface, the system automatically recalculates the geometric correction parameters of each projector and updates the spatial coordinates of the rigid body model in sync. When switching scenes, the system can reconstruct the projection space in real time and update the motion capture mapping rules in sync. The entire process does not require restarting the program.

[0022] Furthermore, the specific steps of generating virtual scene effects based on the spatial motion data and the electromyographic signals using the motion capture mapping rules, and driving the adjustment of lighting and color tones and the regulation of environmental parameters based on the lyrics and text data and the physiological state data, include: The central processing module generates virtual scene effects based on the spatial motion data and the intensity of the electromyographic signals, using the motion capture mapping rules preset in the GPU rendering cluster. When the wrist swing angle is >60° and lasts for 0.5 seconds, particle effects are triggered. When the finger opening and closing frequency is >2 times / second, the scene depth parameters are adjusted. The environmental control device uses the color palette of the DLP projector to adjust the scene lighting and shadows based on the lyrics and physiological state data. It adjusts the temperature within the range of 15-30℃ using the temperature control device and simulates airflow with a wind speed of 0-5m / s using the airflow generator. When it detects that the user's heart rate variability has decreased by 30% for 2 minutes, it triggers synchronous adjustment of temperature and humidity.

[0023] Understandably, based on the spatial motion data and the intensity of the electromyographic signals, virtual scene effects are generated through motion capture mapping rules. Based on the lyrics and the physiological state data, the lighting and color tone are adjusted and the environmental parameters are modified to achieve real-time interaction between user actions and opera content, thereby enhancing the user's physiological immersion.

[0024] Furthermore, the specific steps of activating branch plots and calling cultural knowledge graph related content based on user gaze duration and action sequence include: When the infrared camera tracks the user's gaze at the virtual prop for more than 15 seconds, the associated side story is activated, and the cultural knowledge graph information is displayed through the laser holographic device. When a continuous squatting and forward pushing sequence is detected, the story jumps to the designated scene.

[0025] Understandably, decision trees use user behavior as input nodes and generate different story branches through AI algorithms, thereby supporting personalized user experiences.

[0026] Furthermore, the cultural knowledge graph information includes: Entity layer: 3D models of Kunqu Opera characters and virtual props; Relationship layer: The ownership relationship between props and the script; Attribute layer: vocal style analysis data, Ming Dynasty clothing system text.

[0027] It is understandable that embedding cultural knowledge graphs ensures that innovative interpretations do not deviate from the essence of Kunqu Opera culture.

[0028] In summary, the immersive Kunqu Opera experience method based on a multimodal interaction CAVE system in the above embodiments of the present invention improves the flexibility of the CAVE system by adjusting the CAVE projection space parameters through a visual editing interface and simultaneously updating the motion capture mapping rules. This supports real-time scene switching and allows for real-time reconstruction of the projection space while simultaneously updating the motion capture mapping rules. Based on the spatial motion data and electromyographic signals, virtual scene effects are generated through the motion capture mapping rules. Based on the lyrics and physiological state data, the system drives the adjustment of light and shadow tones and environmental parameters, thereby achieving real-time interaction between the user's natural movements and the projected content and enhancing the user's physiological immersion. Based on the user's gaze duration and action sequence, an AI decision tree algorithm activates branch storylines and calls upon related content from a cultural knowledge graph, thereby supporting personalized user experiences and improving the interactivity between the user and the digital images.

[0029] Example 2: Please see Figure 2 The second embodiment of the present invention also provides an immersive Kunqu Opera experience system 40 based on multimodal interaction CAVE system, including... Editing module 11: Used to adjust CAVE projection space parameters based on the visual editing interface and to update motion capture mapping rules synchronously; Projection module 12: used to load scene files based on the CAVE projection space, project holographic images of characters through a laser holographic device, and generate dynamic backgrounds using a DLP projector; Interaction module 13: used to capture user spatial motion data through an infrared camera, collect user gesture electromyography signals through an electromyography sensor, parse the text data of the lyrics sung by the user through a voice recognition device, and monitor the user's physiological state data through a heart rate sensor. Central processing module 14: used to generate virtual scene effects according to the spatial motion data and the electromyographic signals through the motion capture mapping rules, and to drive the adjustment of light and shadow color tone and the adjustment of environmental parameters according to the lyrics and the physiological state data; Narrative Module 15: Used to activate branch plots based on the user's gaze duration and action sequence, and call cultural knowledge graph related content.

[0030] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for immersive Kunqu Opera experience using a CAVE system based on multimodal interaction, characterized in that, Includes the following steps: Adjust CAVE projection space parameters based on the visual editing interface and update motion capture mapping rules simultaneously; The scene file is loaded based on the CAVE projection space, and the holographic image of the character is projected through a laser holographic device, while a dynamic background is generated using a DLP projector. The system captures user spatial motion data using an infrared camera, collects user gesture electromyography signals using an electromyography sensor, analyzes the text data of the lyrics sung by the user using a voice recognition device, and monitors the user's physiological state data using a heart rate sensor. Based on the spatial motion data and the electromyographic signals, virtual scene effects are generated through the motion capture mapping rules, and the lighting and color tone are adjusted and the environmental parameters are modified based on the lyrics and the physiological state data. Based on the user's gaze duration and action sequence, branch plots are activated through an AI decision tree algorithm, and related content is invoked from a cultural knowledge graph.

2. The immersive Kunqu Opera experience method based on multimodal interaction in a CAVE system according to claim 1, characterized in that, The specific steps for adjusting CAVE projection space parameters and synchronously updating motion capture mapping rules based on the visual editing interface include: In the UnityUGUI editing interface, ScriptableObject is used to achieve two-way data binding between the physical screen model and the motion capture rigid body model; The number of projection surfaces and their spatial coordinates are defined using the physical screen model. The key point coordinates are defined using the motion capture rigid body model, and the motion capture mapping rules are configured.

3. The immersive Kunqu Opera experience method based on multimodal interaction in a CAVE system according to claim 1, characterized in that, The specific steps of generating virtual scene effects based on the spatial motion data and the electromyographic signals using the motion capture mapping rules, and driving the adjustment of lighting and color tones and environmental parameters based on the lyrics and the physiological state data, include: The central processing module generates virtual scene effects based on the spatial motion data and the intensity of the electromyographic signals, using the motion capture mapping rules preset in the GPU rendering cluster. When the wrist swing angle is >60° and lasts for 0.5 seconds, particle effects are triggered. When the finger opening and closing frequency is >2 times / second, the scene depth parameters are adjusted. The environmental control device uses the color palette of the DLP projector to adjust the scene lighting and shadows based on the lyrics and physiological state data. It adjusts the temperature within the range of 15-30℃ using the temperature control device and simulates airflow with a wind speed of 0-5m / s using the airflow generator. When it detects that the user's heart rate variability has decreased by 30% for 2 minutes, it triggers synchronous adjustment of temperature and humidity.

4. The immersive Kunqu Opera experience method based on multimodal interaction in a CAVE system according to claim 1, characterized in that, The specific steps for activating branch plots and calling related content from the cultural knowledge graph based on the user's gaze duration and action sequence include: When the infrared camera tracks the user's gaze at the virtual prop for more than 15 seconds, the associated side story is activated, and the cultural knowledge graph information is displayed through the laser holographic device. When a continuous squatting and forward pushing sequence is detected, the story jumps to the designated scene.

5. The immersive Kunqu Opera experience method based on multimodal interaction in a CAVE system according to claim 1, characterized in that, The cultural knowledge graph information includes: Entity layer: 3D models of Kunqu Opera characters and virtual props; Relationship layer: The ownership relationship between props and the script; Attribute layer: vocal style analysis data, Ming Dynasty clothing system text.

6. An immersive Kunqu Opera experience system based on multimodal interaction using a CAVE system, characterized in that: include: Editing module: Used to adjust CAVE projection space parameters based on a visual editing interface and to update motion capture mapping rules synchronously; Projection module: used to load scene files based on the CAVE projection space, project holographic images of characters through a laser holographic device, and generate dynamic backgrounds using a DLP projector; Interaction module: used to capture user spatial motion data through infrared camera, collect user gesture electromyography signals through electromyography sensor, parse user chanted lyrics text data through voice recognition device, and monitor user physiological state data through heart rate sensor; Central processing module: Based on the spatial motion data and the electromyographic signals, it generates virtual scene effects through the motion capture mapping rules, and drives the adjustment of light and shadow tones and environmental parameters based on the lyrics and the physiological state data; Narrative module: Based on the user's gaze duration and action sequence, branch plots are activated through AI decision tree algorithm, and related content is invoked from cultural knowledge graph.

Images