Immersive stage performance smart subtitle device

Abstract

This smart subtitling system provides an immersive stage experience that addresses the lack of support for hearing-impaired, dyslexic, and multilingual audiences in theaters, while further enhancing the interactive integration of subtitles with the performance and improving the audience's artistic experience and emotional engagement. [Solution] The stage emotion-immersive smart subtitling device 100, applicable to theatrical performance fields, includes: a creative team input side that receives language settings, emotions, and cognitive needs from creative team members; a smart subtitling control center unit connected to the creative team input side that translates based on the subtitling language type selected and input by the creative team and processes subtitle content and emotion annotations; an emotion detection / action recognition module connected to the smart subtitling control center unit that determines the performance state based on a pre-configured performance action and emotion database; a dynamic adaptive font processing module connected to the emotion detection / action recognition module that performs dyslexia-friendly font conversion, personalized layout, and dynamic color adjustment based on the expression needs and content situation of the work; a low-latency synchronous transmission unit that synchronously transmits subtitle information and stage lighting, sound effects, and performance actions; and a stage real-time display device connected to the low-latency synchronous transmission unit that displays smart subtitles integrated with the performance.

Description

Technical Field

[0006]

[0001] The present invention relates to smart assistive technology applied to theater performances, and particularly to a stage emotion immersion type smart subtitle device that interacts with live performances to provide a multilingual and barrier-free viewing experience, catering to the theater viewing needs of the hearing-impaired, dyslexic, as well as audiences of different languages and cultures.

Background Art

[0002] Conventional theater subtitle technologies are commonly seen in opera, musicals, foreign language plays, etc., and only provide text translation services. The static subtitle display method fixed above or on both sides of the stage is common. However, for the hearing-impaired and dyslexic audiences, the lines alone cannot convey the emotions, intonations, and rhythms of the performance, making it difficult to obtain a complete artistic experience.

[0003] Furthermore, most existing subtitles use standard fonts and layouts without adaptation design for dyslexic people, increasing the reading burden and affecting the theater experience. On the other hand, conventional subtitle systems lack real-time interaction with stage elements such as lighting, stage equipment, and the movements of performers, resulting in a mismatch between subtitles and the performance, compromising the overall immersion and artistic aesthetic.

[0004] According to the International Performing Arts Council (IPAC), many hearing-impaired audiences cannot understand the emotions and delicate nuances of the performance even when using subtitle devices, and cannot effectively support the participation of multilingual audiences and cultural exchanges.

[0005] Based on the above, there is a need for a stage emotion immersion type smart subtitle device that solves the above problems and improves the audience's immersion and emotional understanding ability.

[0006] Therefore, based on the current situation where the problems of the prior art have not been effectively solved or overcome, the inventor of the present application files this utility model to solve the above problems.

Summary of the Invention

[0007] The use of the quantifiers "one" or "one" with respect to the components and parts described in this utility model specification is for convenience and to provide the usual meaning of the scope of this utility model, and should be interpreted as including one or at least one in this utility model, and unless otherwise clearly intended, the singular form also includes the plural form.

[0008] The main objective of this utility model is to provide a stage-immersive smart subtitling device that addresses the challenges of conventional subtitling technology in theatrical applications, such as the lack of support for hearing-impaired, dyslexic, and multilingual audiences, while further enhancing the interactive integration of subtitles and performance to improve the artistic experience and emotional participation of the audience.

[0009] Specifically, the technologies employed in this utility model are as follows: The present invention provides a stage immersive smart subtitle system applicable to theatrical performances, comprising: a creative team input side that receives language settings and audience cognitive needs from the creative team; a smart subtitle control center unit connected to the creative team input side that performs translation based on the subtitle language type selected and input by the creative team and processes subtitle content and emotional annotations; an emotion detection / action recognition module connected to the smart subtitle control center unit that determines the performance state based on a pre-configured performance action and emotion database; a dynamic adaptive font processing module connected to the emotion detection / action recognition module that performs dyslexia-friendly font conversion, personalized layout, and dynamic color adjustment based on the expressive needs and content situation of the work; a low-latency synchronous transmission unit that synchronously transmits subtitle information and stage lighting, sound effects, and performance actions; and a stage real-time display device connected to the low-latency synchronous transmission unit that displays smart subtitles integrated with the performance. [Means for solving the problem]

[0010] To achieve the above objectives as a technical means for solving the problem, this utility model provides a stage immersive smart subtitle device applicable to the theater performance field, which includes: a creative team input side that receives language settings and audience cognitive needs from the creative team; a smart subtitle control center unit connected to the creative team input side that performs translation based on the subtitle language type selected and input by the creative team and processes subtitle content and emotional annotations; an emotion detection / action recognition module connected to the smart subtitle control center unit that determines the performance state based on a pre-configured performance action and emotion database; a dynamic adaptive font processing module connected to the emotion detection / action recognition module that performs dyslexia-friendly font conversion, personalized layout, and dynamic color adjustment based on the expression needs and content situation of the work; a low-latency synchronous transmission unit that synchronously transmits subtitle information and stage lighting, sound effects, and performance actions; and a stage real-time display device connected to the low-latency synchronous transmission unit that displays smart subtitles integrated with the performance.

[0011] In the embodiment of this utility model, the smart subtitle control center unit provides emotional annotations for subtitles based on the performer's tone, intonation, and emotion.

[0012] In the embodiment of this utility model, the emotion detection and motion recognition module includes a machine learning algorithm trained to identify motion and voice data during a rehearsal period.

[0013] In the embodiment of this utility model, the dynamically adaptive font processing module uses a font designed for people with dyslexia.

[0014] In the embodiment of this utility model, the dynamically adaptive font processing module adjusts the font weight, size, or color contrast based on the semantic importance of the content.

[0015] In the embodiment of this utility model, the low-latency synchronous transmission unit has a data transmission capability with a delay of less than 50 milliseconds, and the low-latency synchronous transmission unit enables data exchange through 5G, Wi-Fi® 6, or Bluetooth® 5.0 or higher standards.

[0016] In the embodiment of this utility model, the stage real-time display device includes multiple block display modules and can automatically change the subtitle display position according to the performance scene, and the stage real-time display device employs a transparent OLED or projection type display module to reduce visual obstruction.

[0017] In the embodiment of this utility model, the smart subtitle control center unit supports the translation output of synchronized subtitles.

[0018] In the embodiment of this utility model, the subtitles displayed on the real-time stage display device include performance-related instructions other than dialogue (emotional descriptions, ambient sound instructions, and bodily movement notes).

[0019] In the embodiment of this utility model, the stage-immersive smart subtitling device is applicable to theaters, concerts, dance performances, operas, immersive installation art, and other live performance fields. [Effects of the Invention]

[0020] Based on the above explanation, this utility model has the following advantages:

[0021] 1. A multi-sensual, synchronized, and immersive viewing experience. This utility model not only displays the content of the dialogue but also integrates auxiliary information such as emotion, tone, voice, and actions, making subtitles not merely supplementary information but an integral part of the story experience. The audience can simultaneously perceive the rhythm of the voice and the emotions of the performers, significantly enhancing the appeal and immersion of the play.

[0022] 2. Improved readability through dynamic adjustment of layout and fonts. With dyslexia-friendly fonts (e.g., OpenDyslexic) and dynamic character spacing, character color contrast, and line spacing adjustment mechanisms, the subtitles automatically change according to user needs and the story development, showing high adaptability to people with dyslexia, the elderly viewers, and visually impaired viewers, and improving comprehension and comfort.

[0023] 3. Smart emotion and movement cues The subtitles provide supplementary cues such as "excited", "whispered", "approaching uneasily" according to the performance intonation, voice pitch, and body language of the performers, enabling viewers with hearing impairments to understand non-verbal story changes and compensating for the lack of auditory information.

[0024] 4. Accurate synchronization of stage performances through real-time low-latency transmission By realizing a signal transmission delay of less than 50 ms through technologies such as Wi-Fi6, Bluetooth5.0, or 5G, the subtitles are accurately synchronized with the movements of the performers, BGM, and lighting changes, without delay or deviation, far exceeding the existing static subtitle systems.

[0025] 5. Interactive and predictable control system This device has an audience input side and a pre-set story rehearsal database, and can automatically learn and predict the flow of the performance. By preloading the corresponding subtitle information, the response speed is improved, and it can also handle sudden performance changes.

[0026] 6. Advanced integration of subtitles and stage design The subtitle display device can adopt transparent OLED, projector, or floating visual module. The subtitles are linked to the lighting and dynamic arrangement of the scene and naturally blend into the stage, avoiding a sense of incongruity and blocking the performers without damaging the artistic composition.

[0027] 7. Dynamic display and floating subtitle technology Unlike conventional fixed-position subtitles, this invention displays subtitles floating near the actors and scenes in accordance with the story's progression, closely following the flow of the performance, creating a natural flow of the viewer's gaze, and enhancing the audience's "emotional immersion" without obstructing their field of vision.

[0028] 8. Promotion of artistic equality and cultural dissemination The design of this device takes into account the diverse needs of audiences, including those with hearing impairments, dyslexia, and non-native language speakers, and by providing equitable opportunities for art appreciation, it promotes barrier-free participation in theaters and the practice of cultural equality, thereby making a significant social contribution.

[0029] 9. Applicability to various performance and fields In addition to theatrical stages, the technology of this utility model can also be applied to open-air theaters, concerts, operas, immersive exhibits, dance performances, educational demonstrations, museum guides, and more, possessing high applicability and commercial potential. [Brief explanation of the drawing]

[0030] [Figure 1] This is a block diagram of the smart subtitling device for immersive stage performances related to this utility model. [Figure 2] This diagram shows the interaction and synchronization between the subtitles and the stage in this utility model. [Modes for carrying out the invention]

[0031] The embodiments of this utility model will be described in detail below with reference to the drawings. The drawings used herein are for illustrative purposes and supplementary purposes to the specification, and do not represent the actual proportions or precise arrangements after the implementation of this utility model. Accordingly, it goes without saying that the scope of claims in the actual implementation of this utility model should not be limited by the proportions or arrangements shown in the attached drawings.

[0032] Figure 1 is a block diagram of the stage immersion smart subtitling device according to this utility model. Figure 2 is a diagram showing the interaction and synchronization between the subtitles and the stage according to this utility model.

[0033] As shown in Figures 1 and 2, an embodiment of this utility model provides a stage immersive smart subtitle device 100 applicable to the theater performance field, which includes a creative team input side 110 that receives language settings from the creative team and the cognitive needs of the audience; a smart subtitle control center unit 120 connected to the creative team input side 110 that performs translation based on the subtitle language type selected and input by the creative team and processes the subtitle content and emotion annotations; an emotion detection / action recognition module 130 connected to the smart subtitle control center unit 120 that determines the performance state based on a pre-configured performance action and emotion database; a dynamic adaptive font processing module 140 connected to the emotion detection / action recognition module 130 that performs dyslexia-friendly font conversion, personalized layout, and dynamic color adjustment based on the expression needs and content situation of the work; a low-latency synchronous transmission unit 150 that synchronously transmits subtitle information and stage lighting, sound effects, and performance actions; and a stage real-time display device 160 connected to the low-latency synchronous transmission unit 150 that displays smart subtitles integrated with the performance.

[0034] As shown in Figures 1 and 2, in the embodiment of this utility model, the smart subtitle control center unit 120 adds emotional annotations to the subtitles based on the tone, intonation, and emotion of the performer.

[0035] As shown in Figures 1 and 2, in the embodiment of this utility model, the emotion detection and motion recognition module 130 includes a machine learning algorithm trained to identify motion and voice data during the rehearsal period.

[0036] As shown in Figures 1 and 2, in the embodiment of this utility model, the dynamically adaptive font processing module 140 uses a font designed for people with dyslexia.

[0037] As shown in Figures 1 and 2, in the embodiment of this utility model, the dynamically adaptive font processing module 140 adjusts the font weight, size, or color contrast based on the semantic importance of the content.

[0038] As shown in Figures 1 and 2, in the embodiment of this utility model, the low-latency synchronous transmission unit 150 has a data transmission capability with a delay of less than 50 milliseconds, and the low-latency synchronous transmission unit 150 enables data exchange through 5G, Wi-Fi 6, or Bluetooth 5.0 or higher standards.

[0039] As shown in Figures 1 and 2, in the embodiment of this utility model, the stage real-time display device 160 includes multiple block display modules, can automatically change the subtitle display position according to the performance scene, and employs a transparent OLED or projection display module to reduce visual obstruction. Furthermore, by combining it with Meta's (registered trademark) SAM2 (SegmentAnythingModel2), it automatically divides and tracks video / movie objects based on the script and the emotions that the creator wants to express, and can be used to create special effects such as background blurring and object division with simple and real-time operation.

[0040] As shown in Figures 1 and 2, in the embodiment of this utility model, the smart subtitle control center unit 120 supports the translation output of synchronized subtitles.

[0041] As shown in Figures 1 and 2, in the embodiment of this utility model, the subtitles displayed on the stage real-time display device 160 include performance-related instructions other than dialogue (emotional descriptions, ambient sound instructions, and physical movement notes). The subtitles are displayed dynamically in accordance with the performance actions, lighting, and sound effects, enhancing the audience's emotional immersion.

[0042] As shown in Figures 1 and 2, in the embodiment of this utility model, the stage immersion smart subtitling device 100 is applicable to theaters, concerts, dance performances, operas, immersive installation art, and other live performance fields.

[0043] As shown in Figures 1 and 2, in the embodiment of this utility model, the stage emotion-immersive smart subtitling device 100 is installed in the theater space, and the creative team can input parameters such as language (Chinese, English, Japanese, etc.), reading mode (standard, high contrast, dyslexia-friendly font), and speech rate through the device. During the performance, the stage emotion-immersive smart subtitling device 100 detects the actions and voice changes of performer A through the emotion detection / motion recognition module 130 (integrating a camera, acoustic sensor, and AI model) based on pre-set scripts and rehearsal video data, and estimates corresponding emotion annotations and directorial intentions. Furthermore, the smart subtitling control center unit 120 generates subtitle content including story dialogue, tone annotations, and emotion instructions based on this, and adjusts the display style such as font, size, color, and line spacing through the dynamically adaptive font processing module 140. The subtitle information is transmitted via the low-latency synchronous transmission unit 150 (Wi-Fi 6, Bluetooth 5.0, or 5G module, etc.) to a device on the stage 200 equipped with transparent projection display or OLED display technology. The subtitles are displayed in different areas on the stage 200 or move in a floating format in conjunction with the movements of performer A. The stage immersion smart subtitle device 100 automatically adjusts the subtitle display time and duration according to the story development of the performance, maintaining consistency in the viewing rhythm.

[0044] As shown in Figures 1 and 2, in the embodiment of this utility model, the stage-immersive smart subtitling device 100 is applicable to an immersive subtitling display system for concerts. In large concert venues, especially when foreign language songs, fast-paced songs, or the venue environment is noisy, audiences often have difficulty hearing the lyrics. The following effects can be obtained by applying the device 100 of the present invention. Synchronized display of lyrics and emotions: Based on pre-edited lyrics and audio track information, the subtitles recognize and display the lyrics in real time in sync with the live performance, not only displaying the lyrics but also synchronizing and annotating the singer's emotional nuances (e.g., outburst, exhilaration, whisper). Rhythm Interaction Mode: The subtitle font bounces, changes color, and fades in / out in sync with the music's rhythm, enhancing interactive immersion and allowing the audience to not only "read" the lyrics but also "feel" the music's rhythm. Multi-angle display within a given area: Subtitles in different languages ​​and visual styles can be displayed synchronously using multiple devices such as large LED walls, mobile devices, and augmented reality (AR) glasses.

[0045] As shown in Figures 1 and 2, in the embodiment of this utility model, the stage emotion-immersion smart subtitling device 100 is applicable to multilingual subtitling systems for traditional opera or contemporary music theaters. In multilingual cultural exchange settings (such as international opera productions), there are significant differences in audience language comprehension abilities, making it difficult for audiences to fully understand the artistic performance with conventional language subtitles. The device 100 of the present invention can implement the following mechanisms to address this situation: The creative team provides multilingual versions such as Chinese, English, and the original text for audiences of different languages. Emotion translation module: By synchronously processing the translation of tone, rhetoric, and cultural metaphors in language conversion, the audience can not only understand the literal meaning but also experience the artistic emotion. High-resolution transparent subtitle display: In accordance with the stage design, transparent OLED display technology is used to project subtitles above the performers, avoiding obstructing the performers or detracting from the aesthetics of the stage background.

[0046] As shown in Figures 1 and 2, in the embodiment of this utility model, the stage immersion smart subtitling device 100 is applicable to open-air theaters and mobile stages. In open-air theaters, audiences often have difficulty understanding the story due to light source interference, limited viewing angles, and sound diffusion. In this field, the present invention includes the following applications: High-contrast automatic subtitle adjustment: The subtitle module automatically changes brightness and character color according to ambient light levels and temporal changes (daytime, dusk, nighttime, etc.) to ensure visibility. Portable display device: A foldable or magnetically attached display panel is provided and installed in the audience seating area or on the stage veranda. Furthermore, a wearable projection device can be applied to facilitate transport and installation outdoors. Wireless group synchronization technology: Subtitle synchronization between devices is achieved using a Mesh Network, ensuring efficient operation even without a fixed network.

[0047] As shown in Figures 1 and 2, in the embodiment of this utility model, the stage emotion-immersive smart subtitling device 100 is applicable to immersive theaters or improvisational performances. In an immersive performance space, the audience B and performer A are in the same area, there is a high degree of freedom in interaction, and scene changes are frequent, so conventional fixed-position subtitles cannot meet the demands of this type of performance. Therefore, the present invention provides a performer-controlled subtitle trigger system in which performers activate specific subtitle scenarios (such as "shock," "shout," or "hypnosis") by gesture, voice, or sensors to enhance the interactive story atmosphere, and a story-branching subtitle management module that switches between different subtitle versions or branching points according to the audience's interaction path at the performance site, and is applicable to performances with multiple story versions or endings.

[0048] Through the structure described above, this utility model provides the following functions. Deeper emotional communication: Subtitles not only display the dialogue but also note the tone, emotion, and actions, helping the audience understand the story's progression. Readability: Dynamic layout and dyslexia-friendly font design improve reading efficiency and comfort. Real-time interaction integration: Subtitles synchronize with the visuals to enhance visual immersion. Multilingual support: Provides multilingual subtitle output to expand cultural participation. Artistic Unity: The visual design of the subtitles harmonizes with the stage background and lighting, avoiding any sense of incongruity or interference. Its value is clear in terms of its inventiveness, practicality, and ability to meet user needs.

[0049] As described above, the structural components of this utility model are not found in any existing literature or publicly available use, and therefore meet the requirements for filing a utility model application. We respectfully request your review and approval as soon as possible.

[0050] The above describes specific embodiments and the technical principles used in this utility model. It goes without saying that any modifications based on the concept of this utility model fall within the scope of the claims of this utility model, as long as the resulting functions do not exceed the scope of the description in the specification and drawings. [Explanation of Symbols]

[0051] 100 Stage Immersion Smart Subtitle Device 110 Creative Team Input Side 120 Smart Subtitle Control Center Unit 130 Emotion Detection and Motion Recognition Module 140 Dynamically Adaptive Font Processing Module 150 Low-Latency Synchronous Transmission Unit 160 Stage Real-Time Display Device 200 Stage A Performer B Audience

Claims

1. A stage-oriented, emotionally immersive, smart subtitling device applicable to theatrical performances, A creative team inputs language settings and audience cognitive needs, A smart subtitle control center unit is connected to the input side of the creative team, and based on the subtitle language type selected and input by the creative team, it translates the language of the performers, generates subtitle content, and adds emotional annotations. A single emotion detection and motion recognition module connected to the aforementioned smart subtitle control center unit, which determines the performance state based on a pre-configured performance action and emotion database, A dynamic adaptive font processing module is connected to the aforementioned emotion detection and motion recognition module and performs dyslexia-friendly font conversion, personalized layout, and dynamic color adjustment based on the expressive needs and content of the artwork. A single low-latency synchronized transmission unit that synchronizes the transmission of subtitle information with stage lighting, sound effects, and performance actions, and Includes a stage real-time display device connected to the low-latency synchronous transmission unit, which displays smart subtitles integrated with the performance, A smart subtitling device that allows for immersive stage performances.

2. The smart subtitle control center unit is characterized by providing emotional annotations to the subtitles based on the tone, intonation, and emotions of the performers, as described in claim 1.

3. The stage emotion-immersive smart subtitling device according to claim 1, characterized in that the emotion detection and motion recognition module includes a machine learning algorithm trained to identify motion and audio data during the rehearsal period.

4. The stage immersion smart subtitling device according to claim 1, characterized in that the dynamic adaptive font processing module uses a font designed for people with dyslexia.

5. The dynamic adaptive font processing module is characterized by adjusting the font weight, size, or color contrast based on the semantic importance of the content, as described in claim 1.

6. The low-latency synchronous transmission unit is equipped with a data transmission capability with a delay of less than 50 milliseconds, and the low-latency synchronous transmission unit enables data exchange through a standard of 5G, Wi-Fi® 6, or Bluetooth® 5.0 or higher, as described in claim 1.

7. The stage real-time display device includes multiple block display modules and can automatically change the subtitle display position according to the performance scene, and the stage real-time display device employs a transparent OLED or projection display module to reduce visual obstruction, as described in claim 1.

8. The stage-immersion type smart subtitle device according to claim 1, characterized in that the smart subtitle control center unit outputs a translation of synchronized subtitles.

9. The stage immersion smart subtitling device according to claim 1, characterized in that the subtitle content displayed on the real-time stage display device includes direction-related instructions other than dialogue, and the direction-related instructions include emotional descriptions, environmental sound instructions, and physical movement notes.

10. The stage emotional immersion smart subtitling device according to claim 1, characterized in that the aforementioned theatrical performance field includes theater, concerts, dance performances, opera, immersive installation art, and other live performance fields.

Images