Real-time sign language-subtitle-voice three-dimensional synchronous generation system for barrier-free drama

Abstract

The invention relates to the technical field of computer vision and natural language processing, in particular to a barrier-free drama-oriented real-time sign language-subtitle-voice three-dimensional synchronous generation system. Comprising a multi-mode drama content collection module, a drama semantic and emotion deep analysis module, a three-dimensional emotional content generation module, a millisecond-level synchronous calibration module, a personalized demand adaptation module, a multi-terminal output module and a feedback iteration module which are linked in sequence. The multi-mode drama content acquisition module supports offline theaters and online live broadcast / recorded broadcast scenes and can acquire line audios, actor performance data and scene auxiliary data, the deep binding of role personalization, plot emotion, scene atmosphere and barrier-free content is realized for the first time, the pain points of'action stiffness and emotion missing 'of a general system are solved, and the system has the advantages of being high in practicability and high in practicability. The sign language / voice / subtitle drama adaptation degree is improved to 95% or above, the overall time delay is controlled within 100 ms through a synchronous calibration algorithm and is far lower than the standard of 200 ms in the industry, and the drama continuity is guaranteed.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing technology, specifically to a real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible theater. Background Technology

[0002] With the advancement of cultural inclusion policies, accessibility in theatrical arts has become a core requirement for safeguarding the cultural rights of special groups—hearing-impaired individuals rely on sign language or subtitles to understand the plot, while visually impaired individuals require emotional audio narration to aid in perceiving the scenes. However, existing accessible theatrical support systems face numerous technical bottlenecks:

[0003] Multimodal content is out of sync: Sign language generation, subtitle display, and voice broadcast have large latency differences (generally exceeding 200ms) and are not adapted to the rhythm of the drama dialogue and the performance rhythm of the actors, which disrupts the continuity of the plot.

[0004] Sign language generation lacks theatrical adaptability: the general sign language synthesis model outputs stiff movements that cannot match the character settings of theatrical roles (such as the slow sign language of the elderly and the agile sign language of the young) and the emotions of the lines (such as the abrupt sign language of grief and anger and the expansive sign language of joy), and does not support theatrical sign language vocabulary (such as opera jargon and specific titles in period dramas).

[0005] The lack of dramatic flair in the subtitles and audio: the subtitles are simply transcribed text without character annotations or emotional tone markers; the audio is a mechanically synthesized voice, which cannot reproduce the actors' emotional delivery and the atmosphere of the dramatic scene (such as the deep tone in a suspenseful scene or the lighthearted tone in a comedic scene).

[0006] Insufficient personalization: It cannot dynamically adjust output parameters according to the individual needs of special groups (such as the sign language speed preferences of hearing-impaired users and the speech speed requirements of visually impaired users);

[0007] Poor real-time performance and stability: In online drama live streaming scenarios, the system is prone to parsing lag or generation errors when faced with unexpected situations such as broken lines, dialect words, and improvisational performances, which cannot guarantee the continuity of accessibility services. To address this, a real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama is proposed. Summary of the Invention

[0008] In view of this, the present invention provides a real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama, in order to solve or alleviate the technical problems existing in the prior art, and at least provide a beneficial alternative.

[0009] The technical solution of this invention is implemented as follows: a real-time sign language-subtitle-voice three-dimensional synchronous generation system for accessible drama, including a multimodal drama content acquisition module, a drama semantic and emotional depth analysis module, a three-dimensional emotional content generation module, a millisecond-level synchronous calibration module, a personalized demand adaptation module, a multi-terminal output module, and a feedback iteration module linked in sequence.

[0010] The multimodal drama content acquisition module supports offline theaters and online live / recorded broadcast scenarios. It can collect dialogue audio, actor performance data, and scene auxiliary data. It adopts an edge-cloud collaborative architecture with a acquisition latency of ≤50ms.

[0011] The deep analysis module for drama semantics and emotions includes sub-modules for intelligent analysis of drama dialogues, character and emotion association, and background sound effects scene matching. It adopts the BERT-Drama model that integrates a drama dictionary, the GCN character emotion association model, and the CNN-LSTM emotion classification model, which can identify drama-specific vocabulary and dialects and determine the polarity and intensity of emotions.

[0012] The three-dimensional emotional content generation module includes character-based emotional sign language, scene-based intelligent subtitles, and emotional character voice submodules. It adopts the GAN-ASL sign language generation model and the Tacotron2-Drama speech synthesis model, and can generate 3D sign language avatars, multi-dimensional subtitles, emotional voices and scene explanations that match the character's personality and the plot's emotions.

[0013] The millisecond-level synchronization calibration module can establish a unified timestamp benchmark and control the overall synchronization latency to ≤100ms through a dynamic latency compensation algorithm, while supporting load reduction strategies to ensure stability.

[0014] The personalized needs adaptation module supports customization of sign language speed, avatar style, and subtitle style for hearing-impaired users, and customization of speech speed and narration detail for visually impaired users. It can also save personalized configuration files.

[0015] The multi-terminal output module is compatible with offline theater LED screens / dedicated receivers, online live picture-in-picture / dual audio tracks, mobile terminal APPs and other terminals.

[0016] The feedback iteration module uses reinforcement learning to fine-tune the core model, and can update the drama-specific vocabulary database and the character sign language gesture database.

[0017] More preferably, the data collected by the multimodal drama content acquisition module includes: dialogue audio and background sound effects with a sampling rate of 48kHz, actor body / expression videos, character design tags, plot scene tags, stage lighting / prop switching signals, and supports noise suppression of stage echo and audience noise.

[0018] In a further preferred embodiment, the intelligent drama dialogue parsing submodule has a built-in drama-specific vocabulary library, which can identify specialized vocabulary such as opera jargon, ancient costume titles, and genre terminology, and complete the rhythmic intelligent segmentation of the dialogue through the CTC algorithm.

[0019] In a further preferred embodiment, the character and emotion association submodule constructs a "character-emotion-action" association graph through a GCN model, which can map character settings such as "middle-aged refined scholar" and "fiery street woman" into exclusive emotional features, and output the emotional polarity and intensity by combining plot scene tags.

[0020] Furthermore, the characterized emotional sign language generation submodule has a built-in theatrical character sign language gesture library, which can generate 3D sign language avatars that match the character's personality, map the intensity of emotions to the amplitude and pauses of sign language movements, and support the custom configuration of theatrical-specific sign language vocabulary.

[0021] Furthermore, the scenario-based intelligent subtitle generation submodule can output multi-dimensional subtitles that include dialogue text, character names, emotional markers, and background sound effects. It supports switching subtitle styles according to the scenario, and the subtitle font size is ≥24 points and is a clear sans-serif font.

[0022] More preferably, the dynamic latency compensation algorithm of the millisecond-level synchronization calibration module can monitor the processing latency of each module and achieve compensation through pre-caching and dynamic speed adjustment. The synchronization threshold is ≤100ms. When the threshold is exceeded, a load reduction strategy can be triggered to simplify sign language details.

[0023] Furthermore, the emotional character voice generation submodule can generate emotional voices that match the character's timbre, while providing visually impaired users with scene narration voices that include actor movements and lighting changes, and supports dual-track output of "pure dialogue" and "dialogue + narration".

[0024] The embodiments of the present invention have the following advantages due to the adoption of the above technical solutions:

[0025] I. This invention is the first to achieve deep integration of character design, plot emotions, scene atmosphere and accessible content, solving the pain points of "stiff movements and lack of emotion" in general systems. The drama adaptation of sign language / voice / subtitles is improved to over 95%, and the synchronization calibration algorithm controls the overall latency to within 100ms, far below the industry standard of 200ms, ensuring the continuity of the plot.

[0026] Second, this invention supports the customized needs of hearing-impaired / visually-impaired groups, while adapting to multiple offline / online scenarios and covering different viewing channels. The edge-cloud collaborative architecture ensures low latency in data collection and processing. It has a built-in drama-specific dictionary and model, which can handle complex situations such as dialects, slang, and improvisational performances. The feedback iteration module realizes dynamic optimization of models and functions, continuously improving the accuracy and user experience of accessibility services.

[0027] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the invention will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0030] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0031] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] like Figure 1 As shown, this embodiment of the invention provides a real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama, including a multimodal drama content acquisition module, a drama semantic and emotional depth analysis module, a three-dimensional emotional content generation module, a millisecond-level synchronization calibration module, a personalized needs adaptation module, a multi-terminal output module, and a feedback iteration module linked in sequence.

[0033] The multimodal drama content acquisition module supports offline theaters and online live / recorded scenarios. It can acquire dialogue audio, actor performance data, and scene auxiliary data. It adopts an edge-cloud collaborative architecture with an acquisition latency of ≤50ms.

[0034] The deep analysis module of drama semantics and emotions includes sub-modules for intelligent analysis of drama lines, character and emotion association, and background sound effect scene matching. It adopts the BERT-Drama model that integrates drama dictionary, GCN character emotion association model, and CNN-LSTM emotion classification model, which can identify drama-specific vocabulary and dialects and complete the determination of emotion polarity and intensity.

[0035] The 3D emotional content generation module includes character-based emotional sign language, scene-based intelligent subtitles, and emotional character voice submodules. It adopts the GAN-ASL sign language generation model and the Tacotron2-Drama speech synthesis model, which can generate 3D sign language avatars, multi-dimensional subtitles, emotional voices, and scene explanations that match the character's personality and the plot's emotions.

[0036] The millisecond-level synchronization calibration module can establish a unified timestamp benchmark and control the overall synchronization latency to ≤100ms through a dynamic latency compensation algorithm, while supporting load reduction strategies to ensure stability.

[0037] The personalized needs adaptation module supports customization of sign language speed, avatar style, and subtitle style for hearing-impaired users, and customization of speech speed and narration detail for visually impaired users. It can also save personalized configuration files.

[0038] Multi-terminal output module, compatible with offline theater LED screens / dedicated receivers, online live picture-in-picture / dual audio tracks, mobile terminal APP and other terminals;

[0039] The feedback iteration module uses reinforcement learning to fine-tune the core model and can update the drama-specific vocabulary and character sign language gesture database.

[0040] In one embodiment, the data collected by the multimodal drama content acquisition module includes: dialogue audio and background sound effects with a sampling rate of 48kHz, actor body / expression videos, character tags, plot scene tags, stage lighting / prop switching signals, and supports noise suppression of stage echo and audience noise.

[0041] In one embodiment, the intelligent drama dialogue parsing submodule has a built-in drama-specific vocabulary library, which can identify specialized vocabulary such as opera jargon, ancient costume titles, and genre terms, and complete the rhythmic intelligent segmentation of the dialogue through the CTC algorithm.

[0042] In one embodiment, the character and emotion association submodule constructs a "character-emotion-action" association graph through the GCN model, which can map character types such as "middle-aged refined scholar" and "fiery street woman" into exclusive emotional characteristics, and output the emotional polarity and intensity by combining plot scene tags.

[0043] In one embodiment, the characterized emotional sign language generation submodule has a built-in theatrical character sign language gesture library, which can generate 3D sign language avatars that match the character's personality, map the intensity of emotions to the amplitude and pauses of sign language movements, and support the custom configuration of theatrical-specific sign language vocabulary.

[0044] In one embodiment, the scenario-based intelligent subtitle generation submodule can output multi-dimensional subtitles that include dialogue text, character names, emotional markers, and background sound effects. It supports switching subtitle styles according to the scenario, and the subtitle font size is ≥24 points and is a clear sans-serif font.

[0045] In one embodiment, the dynamic latency compensation algorithm of the millisecond-level synchronization calibration module can monitor the processing latency of each module and achieve compensation through pre-caching and dynamic speed adjustment. The synchronization threshold is ≤100ms. When the threshold is exceeded, a load reduction strategy can be triggered to simplify sign language details.

[0046] In one embodiment, the emotional character voice generation submodule can generate emotional voices that match the character's timbre, while providing visually impaired users with scene narration voices that include actor movements and lighting changes, and supports dual-track output of "pure dialogue" and "dialogue + narration".

[0047] In one embodiment, an example of accessible services for the third act of the offline stage play "Thunderstorm"

[0048] Content Acquisition: The system collects the dialogue between Zhou Puyuan and Lu Shiping through a lavalier microphone, captures their excited facial expressions and body movements through a stage camera, and obtains the character tags "middle-aged dignified capitalist (Zhou Puyuan)" and "angry lower-class woman (Lu Shiping)" and the scene tag "argument in an old house on a rainy night" through the script interface.

[0049] Semantic sentiment analysis: The BERT-Drama model identifies specific expressions in the dialogue such as "the old story of thirty years ago" and "Zhou Mansion in Wuxi". The CNN-LSTM model determines the sentiment of the dialogue as "anger + sadness". The GCN model maps the character settings as the emotional features of Zhou Puyuan "calm but furious" and Lu Shiping "hurried but aggrieved".

[0050] 3D content generation:

[0051] Sign language generation: GAN-ASL generated dignified sign language with "large amplitude and strong pauses" for Zhou Puyuan and grief-stricken sign language with "rapid and trembling hands" for Lu Shiping, and generated corresponding avatars by matching the characters of the two.

[0052] Subtitle generation: Output multi-dimensional subtitles such as "[Zhou Puyuan (angry)] What exactly do you want! [Lu Shiping (sorrowful)] I just want justice for my daughter! [Background: Thunder and rain intensify]";

[0053] Voice generation: Generate emotional voice and scene narration for visually impaired users, such as "Zhou Puyuan questioned in a stern voice with an angry tone; Lu Shiping's voice choked with sobs, and the sound of dense thunder and rain in the background."

[0054] Synchronization calibration: The system detected a sign language generation delay of 110ms, a subtitle delay of 30ms, and a speech delay of 60ms. By pre-caching subtitles and fine-tuning the speech rate, the overall synchronization delay was controlled to 80ms.

[0055] Personalized adaptation: A hearing-impaired user enabled "sign language slowdown 20%" and "subtitle magnification 15%", while a visually impaired user enabled "voice speed increase 10%" and "advanced scene explanation mode".

[0056] Terminal output: Subtitles and sign language avatars are displayed on the side screen of the stage, and emotional voice and scene explanations are pushed to the receiver of visually impaired users. User feedback is that "the sign language movements fit the character's personality and are synchronized without delay."

[0057] Feedback Iteration: After collecting feedback, the system fine-tunes the trembling amplitude of Lu Shiping's sign language and optimizes the model's emotion mapping accuracy.

[0058] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible theater, characterized by: It includes a multimodal drama content acquisition module, a drama semantic and emotional depth analysis module, a three-dimensional emotional content generation module, a millisecond-level synchronization calibration module, a personalized demand adaptation module, a multi-terminal output module, and a feedback iteration module, all linked in sequence. The multimodal drama content acquisition module supports offline theaters and online live / recorded broadcast scenarios. It can collect dialogue audio, actor performance data, and scene auxiliary data. It adopts an edge-cloud collaborative architecture with a acquisition latency of ≤50ms. The deep analysis module for drama semantics and emotions includes sub-modules for intelligent analysis of drama dialogues, character and emotion association, and background sound effects scene matching. It adopts the BERT-Drama model that integrates a drama dictionary, the GCN character emotion association model, and the CNN-LSTM emotion classification model, which can identify drama-specific vocabulary and dialects and determine the polarity and intensity of emotions. The three-dimensional emotional content generation module includes character-based emotional sign language, scene-based intelligent subtitles, and emotional character voice submodules. It adopts the GAN-ASL sign language generation model and the Tacotron2-Drama speech synthesis model, and can generate 3D sign language avatars, multi-dimensional subtitles, emotional voices and scene explanations that match the character's personality and the plot's emotions. The millisecond-level synchronization calibration module can establish a unified timestamp benchmark and control the overall synchronization latency to ≤100ms through a dynamic latency compensation algorithm, while supporting load reduction strategies to ensure stability. The personalized needs adaptation module supports customization of sign language speed, avatar style, and subtitle style for hearing-impaired users, and customization of speech speed and narration detail for visually impaired users. It can also save personalized configuration files. The multi-terminal output module is compatible with offline theater LED screens / dedicated receivers, online live picture-in-picture / dual audio tracks, mobile terminal APPs and other terminals. The feedback iteration module uses reinforcement learning to fine-tune the core model, and can update the drama-specific vocabulary database and the character sign language gesture database.

2. The real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama according to claim 1, characterized in that: The multimodal drama content acquisition module collects data including: dialogue audio and background sound effects with a sampling rate of 48kHz, actor body / expression videos, character tags, plot scene tags, stage lighting / prop switching signals, and supports noise suppression of stage echo and audience noise.

3. The real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama according to claim 1, characterized in that: The intelligent parsing submodule for drama dialogue has a built-in drama-specific vocabulary library, which can identify specialized vocabulary such as opera jargon, ancient costume titles, and genre terms, and complete the rhythmic intelligent segmentation of the dialogue through the CTC algorithm.

4. The real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama according to claim 1, characterized in that: The character and emotion association submodule constructs a "character-emotion-action" association graph through the GCN model, which can map the character settings such as "middle-aged refined scholar" and "fiery street woman" into exclusive emotional characteristics, and output the emotional polarity and intensity by combining plot scene tags.

5. The real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama according to claim 1, characterized in that: The characterized emotional sign language generation submodule has a built-in theatrical character sign language gesture library, which can generate 3D sign language avatars that match the character's personality, map the intensity of emotions to the amplitude and pauses of sign language movements, and support the custom configuration of theatrical-specific sign language vocabulary.

6. The real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama according to claim 1, characterized in that: The scenario-based intelligent subtitle generation submodule can output multi-dimensional subtitles that include dialogue text, character names, emotional markers, and background sound effects. It supports switching subtitle styles according to the scenario, and the subtitle font size is ≥24 points and is a clear sans-serif font.

7. The real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama according to claim 1, characterized in that: The dynamic latency compensation algorithm of the millisecond-level synchronization calibration module can monitor the processing latency of each module and achieve compensation through pre-caching and dynamic speed adjustment. The synchronization threshold is ≤100ms. When the threshold is exceeded, a load reduction strategy can be triggered to simplify sign language details.

8. The real-time sign language-subtitle-speech three-dimensional synchronous generation system for accessible drama according to claim 1, characterized in that: The emotional character voice generation submodule can generate emotional voices that match the character's timbre, and at the same time provide visually impaired users with scene narration voices that include actor movements and lighting changes, supporting dual audio track output of "pure dialogue" and "dialogue + narration".

Images