Script extraction and subtitle transmission system synchronized with live performing art contents

Abstract

A system providing real-time subtitles during live performances, such as plays, operas and musicals, in a preferred language and to aid the hearing impaired. The display device may be AR or smart glas

Description

Invention Title: SCRIPT EXTRACTION AND SUBTITLE TRANSMISSION SYSTEM SYNCHRONIZED WITH LIVE PERFORMING ART CONTENTS Technical Field [1] The technical idea of the present disclosure relates to a script extraction and subtitle transmission system synchronized with live performing art contents and, more specifically, relates to a script extraction and subtitle transmission system and device synchronized with live performing art contents, for projecting in front of a viewer's eyes commentary subtitles synchronized in real time with performing arts in a preferred language, by using an AR display device when watching verbal performing arts such as plays or musicals . Background Art [2] Unless otherwise indicated in the present specification, the contents described in this section are not a prior art with respect to the claims of the present application, and being included in this section does not mean that it is recognized as a prior art. [3] Performing arts are all forms of arts performed on stage, and are essentially a form of arts where an entity of the work is created on the spot while performers and audience share the same time and space. Nonverbal performances, which remove language barriers in order to overcome the limitations of language barriers in the performing art world, have been a breath of fresh air in the performance market since the end of the 20th century and the popularity is widespread in the United States. However, the performance method for removing the border of languages has been only the nonverbal performance method so far, which restricts performance planners to enter, so there is a need for a service that allows for freely watching the original work with languages while wearing glasses provided with commentary subtitles in a preferred language. [4] Meanwhile, subtitles are an important factor in enhancing the understanding of performing arts and are a major means for improving comprehension of people with hearing impairments and of performances of original works in foreign languages, by resolving barriers in languages different from each other. In addition, it may be helpful even to audiences, who understand the language, to understand the contents of the performance. [5] A subtitle service method for language barriers in the largest performing art venues in Korea and around the world, such as Broadway, West End, and Tokyo Shiki, displays subtitles on screens fixed on both sides and up and down of the stage or on the seats as shown in FIG. 1, which distracts the viewer's attention as well as causes differences in comprehension of the performance depending on the viewer's seating position. [6] In addition, the existing method of providing the service not only causes greater fatigue because of a long movement distance of the sightline between the live performance and the subtitles on the screen, but also makes it difficult to sufficiently recognize performance information and subtitle information because of the distraction of the sightline. In the verbal performing art market around the world, various attempts are being made to solve the proposition of the language barrier, but there is currently no clear solution. Disclosure Technical Problem [7] A script extraction and subtitle transmission device synchronized with live performing art contents according to an exemplary embodiment may project in front of a viewer's eyes commentary subtitles synchronized in real time with performance contents in a preferred language, by using a display device including AR smart subtitle glasses to allow for simultaneously watching together with the performance, by removing language barriers according to nationality or sound barriers caused by deafness or hearing loss, in order to allow for watching the performing arts when watching verbal performance arts such as plays or musicals. [8] In an exemplary embodiment, only the vocals are collected by removing background music from the actor's dialog and music sounds during live performances such as plays, musicals, and operas in the performing art field, and dialog information converted through a Speech-to-Text (STT) AI engine is compared with the original transcript contents to recognize a synchronization point and is mapped with the scenario held in advance, thereby projecting the subtitles onto AR glasses and display devices. [9] In addition, in an exemplary embodiment, each actor's voice is captured by using a microphone. The captured analog signal is transmitted to an individual channel of an audio mixer, and a venue audio mixer mixes the voice channel signals, excluding unnecessary background sound or music channels, and routes the same to an output bus.

[10] In addition, in an exemplary embodiment, the signal from the audio mixer is transmitted to an audio interface and the digital signal is transmitted to the desktop application through USB. In addition, the desktop application segments audio in a unit of chunks and transmits the chunk data to the server.

[11] In addition, in an exemplary embodiment, the server accumulates audio chunks for a certain period of time, inputs the accumulated audio data into the STT model to convert into a text, and performs a search on the five transcript sentences after the previously matched index of the script. In this case, in the exemplary embodiment, the search range is limited to optimize the computation speed.

[12] In addition, in an exemplary embodiment, when a script exceeding a threshold value is found by measuring distancebased similarity, the corresponding dialog is determined as a matched dialog, and the script is transmitted to the desktop application when matching is successful.

[13] In addition, in an exemplary embodiment, the desktop application transmits real-time script data to smart glasses by using a WebSocket protocol through an internal network (local network), and the smart glasses display the received script data on a screen in real time.

[14] However, the tasks to be solved according to an exemplary embodiment are not limited to those mentioned above. Technical Solution

[15] A script extraction and subtitle transmission system synchronized with live performing art contents according to an exemplary embodiment includes a microphone for collecting sound containing the actor's dialog and the music during live performances including plays, musicals, and operas in a performing art field, a server for removing background music from the sound collected from the microphone and filtering only vocals to input into a Speech-to-Text (STT) model, for comparing dialog information converted into a text with an original transcript content to recognize a synchronization point, and for mapping with a scenario stored in advance to transmit subtitles onto an output device according to a mapping result, an audio mixer for receiving the sound as an input to adjust volume, tone, and panning of each signal, and for mixing and transmitting as a single output signal, and an output device for outputting the subtitles by projecting the subtitles received from the server onto a display, and for outputting the output signal.

[16] In addition, the server includes a memory for storing at least one command for a script extraction and subtitle transmission synchronized with live performing art contents, and a processor for performing operations according to the command, wherein the processor captures voice from the sound transmitted from the microphone, and transmits the captured voice to an individual channel of the audio mixer, and the audio mixer can mix and route voice channel signals to an output bus while excluding unnecessary background sound or music channels.

[17] In addition, the processor may receive chunk data, where audio is segmented in a unit of chunks, from a desktop application, accumulates the received audio chunks for a certain period of time, and inputs the accumulated audio data into the STT model to convert into a text.

[18] In addition, the processor may proceed with a search on a certain number of script sentences after a previously matched index of the script, and when a script exceeding a threshold value is detected by measuring distance-based similarity, the detected dialog may be determined as a matching dialog.

[19] In addition, the processor may limit the script sentences of a search range to fewer than a certain number.

[20] In addition, in the event of matching success, the processor may transmit a matched script to the desktop application and initialize the existing audio chunk.

[21] In addition, the desktop application may transmit real-time script data to the output device including smart glasses by using a WebSocket protocol through a local network, and the smart glasses display the received script data on a screen of a display in real time.

[22] In addition, the processor may attempt state transitions according to a threshold while receiving each chunk in order to perform a state machine-based speech chunk analysis, and determine whether to transmit or initialize the dialog after checking transcript similarity according to the current state. Advantageous Effects

[23] A script extraction and subtitle transmission device synchronized with live performing art contents according to an exemplary embodiment can provide subtitles in a language preferred by the audience in performances such as plays, musicals, and operas performed in foreign languages, thereby removing language barriers.

[24] In addition, in an exemplary embodiment, it may be possible to foster an environment, where various audiences can understand the same performance regardless of what language they speak, and to help deaf or hard-of-hearing audiences to understand the dialog of the performance through real-time subtitles, thereby improving accessibility to performance viewing. In addition, by complementarily providing visual information in addition to voice-based information transmission, audiences can see the synchronized dialog in real time in front of their eyes, and can immediately understand the contents of the performance without missing anything.

[25] In addition, in an exemplary embodiment, the script and performance can be synchronized so that the context of the scene can be viewed without missing, and accurate subtitles can be provided in real time through STT AI and scenario mapping.

[26] In addition, in an exemplary embodiment, the accuracy of dialog can be improved by removing unnecessary background music and by utilizing only voice channels, and the reliability of subtitle provision can be improved by matching the optimal dialog through distance-based similarity measurement.

[27] In addition, through an exemplary embodiment, the usability of the audience can be maximized by utilizing display devices such as AR smart subtitle glasses, and real-time data transmission through WebSocket can enable fast and stable data transmission.

[28] In addition, through an exemplary embodiment, it may be possible to process in real time various information generated during the performance so that the actor's impromptu dialog or changed matters in the performance can be quickly reflected in subtitles, and technical latency during the performance may be minimized to maintain the viewing flow. In addition, a new type of experience may be provided for allowing the audience to watch subtitles simultaneously linked to the performance through a smart device.

[29] Furthermore, the original version of performing art contents can be viewed anywhere overseas without language barriers through AR subtitle glasses, and the global performing art industry can enter a new market by providing a service that breaks down language barriers.

[30] In addition, through an exemplary embodiment, AI acoustic feature extraction and voice-subtitle conversion technology can be utilized to provide a subtitle transmission service synchronized with the contents during live performances of performing arts, and a service can be provided that allows audiences to set their preferred language to improve their understanding and immersion in the performance even with language barriers in the original work.

[31] In addition, through an exemplary embodiment, the audience can watch the performance through the worn AR subtitle glasses while simultaneously observing the actor's acting and movement, and have a more comfortable and enjoyable viewing experience. This can not only increase audience satisfaction, but also contribute to improving the popularity and reputation of the performance.

[32] In addition, through an exemplary embodiment, it can be possible to provide performances where performing art contents around the world are presented in localized language expressions .

[33] In addition, since performing arts are contents that attract many audiences of various nationalities and languages, it can be possible to provide a more convenient and comfortable viewing environment for multicultural audiences by introducing subtitle glasses technology. Multicultural audiences can see and understand subtitles translated into their native language, which can provide a more friendly and inclusive performance environment.

[34] In addition, through the subtitle glasses provided in an exemplary embodiment, the audience can watch the performance while simultaneously observing the actor's acting and movement, and have a more comfortable and enjoyable viewing experience. 5 This can not only increase audience satisfaction, but also contribute to improving the popularity and reputation of the performance.

[35] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to the effects mentioned 10 above, and other effects not mentioned can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the following description. That is, unintended effects resulting from practicing the exemplary embodiments of the 15 present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure. Description of Drawings

[36] FIG. 1 is a view showing screen subtitles of a performing art venue currently in use.

[37] FIG. 2 is a view showing an architecture for a script extraction and subtitle transmission service synchronized with live performing art contents according to an exemplary embodiment.

[38] FIG. 3 is a view showing a script extraction and subtitle transmission system synchronized with live performing art contents according to an exemplary embodiment.

[39] FIG. 4 is a view showing a server configuration according to an exemplary embodiment.

[40] FIG. 5 is a view for illustrating a function of a desktop application according to an exemplary embodiment.

[41] FIG. 6 is a view showing a process of displaying the received script data in real time on a screen of a display in smart glasses according to an exemplary embodiment.

[42] FIG. 7 is a view showing a voice chunk state search algorithm according to an exemplary embodiment. Best Mode

[43] A script extraction and subtitle transmission device synchronized with live performing art contents according to an exemplary embodiment may project in front of the viewer's eyes commentary subtitles synchronized in real time with performance contents in a preferred language, by using a display device including AR smart subtitle glasses to allow for simultaneously watching together with the performance, by removing language barriers or sound barriers caused by deafness or hearing loss, in order to allow for watching the performing arts when watching verbal performance arts such as plays or musicals. Mode for Invention

[44] Hereinafter, various exemplary embodiments of the present disclosure are described in connection with the accompanying drawings. Various exemplary embodiments of the present disclosure may be subject to various modifications and may have a variety of exemplary embodiments, and specific exemplary embodiments may be illustrated in the drawings and may be described in detail in connection therewith. However, it is not intended to limit various exemplary embodiments of the present disclosure to specific exemplary embodiments and it should be understood to include all modifications and / or equivalents or substitutes included in the spirit and technical scope of various exemplary embodiments of the present disclosure. With respect to the description of the drawings, similar reference numerals are used for similar components .

[45] In various exemplary embodiments of the present disclosure, terms such as "include", "have", and the like are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[46] In various exemplary embodiments of the present disclosure, expressions such as "or" and the like may include any and all combinations of words listed together. For example, "A or B" may include A, include B, or include both A and B.

[47] Expressions such as "one", "two", "first", or "second" used in various exemplary embodiments of the present disclosure may modify various components of various exemplary embodiments, but do not limit the corresponding components. For example, the expressions above do not limit the order and / or importance of the corresponding components, and may be used to distinguish one component from another component.

[48] When it is said that a component is "connected" or "linked" to another component, it should be understood that while the component may be directly connected to or linked to another component, a new component may exist between the component and another component.

[49] In an exemplary embodiment of the present disclosure, terms such as "module", "unit", "part", and the like may be used to refer to components that perform at least one function or operation, and these components may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", and the like may be implemented as at least one processor integrated into at least one module or chip, except where each needs to be implemented as individual specific hardware.

[50] Terms such as those generally defined in commonly used dictionaries should be interpreted as having a meaning consistent with the contextual meaning of the relevant technology, and are not interpreted in an idealized or overly formal sense unless clearly defined in various exemplary embodiments of the present disclosure.

[51] Hereinafter, an exemplary embodiment of the present disclosure will be described in detail with reference to the accompanying drawings .

[52] FIG. 2 is a view showing an architecture for a script extraction and subtitle transmission service synchronized with live performing art contents according to an exemplary embodiment.

[53] Referring to FIG. 2, a script extraction and subtitle transmission system synchronized with live performing art contents according to an exemplary embodiment may remove background music from the actor's dialog and the music sound during live performances such as plays, musicals, and operas in the performing art field, and only vocals may be collected and inputted to the Speech-to-Text (STT) model. Afterwards, dialog information converted into a text may be compared with the original transcript contents to recognize a synchronization point and may be mapped with the scenario held in advance, thereby projecting the subtitles onto AR glasses and display devices. In an exemplary embodiment, each actor's voice may be captured by using a microphone, and the captured analog signal may be transmitted to an individual channel of the audio mixer. In addition, the audio mixer of the performance venue may mix the voice channel signals, excluding unnecessary background sound or music channels, and may route the same to an output bus. In addition, in an exemplary embodiment, the signal from the audio mixer may be transmitted to an audio interface. In addition, in the exemplary embodiment, digital signals may be transmitted to a desktop application via USB.

[54] FIG. 3 is a view showing a script extraction and subtitle transmission system synchronized with live performing art contents according to an exemplary embodiment. Referring to FIG. 3, the script extraction and subtitle transmission system synchronized with live performing art contents according to an exemplary embodiment may be configured to include a microphone 10, a server 100, an audio mixer 20, and an output device 30. The microphone 10 may collect sound including the actor's dialog and the music during live performances including plays, musicals, and operas in the performing art fields. When only vocals are filtered by removing background music from the sound collected from the microphone 10 and are inputted into the Speech-to-Text (STT) model, the server 100 may compare dialog information converted into a text with the original transcript contents to recognize the synchronization point and may map the same with the scenario held in advance, thereby transmitting the subtitles onto the output device 30 according to the mapping result. The audio mixer 20 may receive sounds as inputs, adjust the volume, tone and panning of each signal, and mix and transmit the same as a single output signal. The output device 30 may project the subtitles received from the server 100 onto a display to output subtitles, and may output an output signal.

[55] FIG. 4 is a view showing a block diagram of a server according to an exemplary embodiment. In an exemplary embodiment, the server may be a computing system for providing services to other computers or devices in a computer network or for storing and managing data. The server 100 may receive a request from other computers or devices called a client, and may provide a response or data for the corresponding request. The configuration of the server 100 shown in FIG. 2 is merely a simplified example.

[56] The communication module 110 may be configured regardless of communication patterns such as wired and wireless, and may be configured as various communication networks such as a personal area network (PAN), a local area network (LAN), a wide area network (WAN), and the like. In addition, the communication module 110 may operate on the basis of the known World Wide Web (WWW) , and may use a wireless transmission technology used for short-range communication such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication module 110 may take charge of transmitting and receiving data necessary to perform techniques according to an exemplary embodiment of the present disclosure.

[57] The memory 120 may refer to any type of storage medium. For example, the memory 120 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. Such a memory 120 may constitute a database shown in FIG. 1.

[58] The memory 120 may store at least one command that is executed by the processor 130. In addition, the memory 120 may store any form of information generated or determined by the processor 130 and any form of information received by the server 200. In addition, the memory 120 may store various types of modules, a set of commands, or models.

[59] The processor 130 may perform technical features according to the exemplary embodiments of the present disclosure to be described later by executing at least one command stored in the memory 120. In an exemplary embodiment, the processor 130 may be configured with at least one core, and may include a processor for data analysis and / or processing, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU) , and a tensor processing unit (TPU) of a computer device.

[60] Such a processor 130 may train a neural network or model designed in a machine learning or deep learning manner. To this end, the processor 130 may perform computations for training the neural network, such as processing input data for training, extracting features from the inputted data, calculating errors, and updating weights of the neural network by using backpropagation. In addition, the processor 130 may perform inference for a predetermined purpose by using a model implemented in an artificial neural network manner.

[61] In an exemplary embodiment, the processor 130 may capture voice from the sound transmitted from the microphone and may transmit the captured voice to an individual channel of the audio mixer. To this end, the processor 130 may receive analogue sound signals transmitted from the microphone worn or used by each actor during the performance. The corresponding signals may include various audio elements such as the actor's dialog, background noise, surrounding music, and the like. Thereafter, the processor 130 may analyze the received sound signal to separate and capture a signal including a voice. For example, the processor 130 may utilize a specific frequency band filtering, a noise removal algorithm, or a voice activity detection technology (VAD). The process described above may be to secure the clarity of the dialog signal. In addition, the processor 130 may convert the captured voice signal into a digital or analog form to transmit to the individual channel of the audio mixer. The signal of each microphone may be assigned to channels different from each other, and thus the audio mixer may independently control and mix each actor's voice. In an exemplary embodiment, the processor 130 may apply sampling rate adjustment, signal compression and restoration, or other audio signal processing technologies so that the quality of the voice can be maintained during the signal transmission process. This may be to secure the stability of real-time processing during the performance.

[62] In addition, in an exemplary embodiment, the audio mixer may mix voice channel signals, excluding unnecessary background sound or music channels, and may route the same to the output bus. In an exemplary embodiment, the audio mixer may receive signals generated from each microphone, music instrument, or background sound during the performance and may assign the same to the individual channel. Each channel may be configured as an independently adjustable signal path, and the actor's voice signal may be identified as a voice channel. Thereafter, the audio mixer may first focus on the frequency band of the voice channel and remove background sound or unnecessary noise in order to exclude a specific channel (background sound, music, etc.) from the inputted signal. In addition, the signal in the channel may be analyzed through the voice activity detection algorithm (VAD) and only the part containing the voice signal may be extracted. Thereafter, signal intensity adjustment may be performed to minimize unintentional noise and improve the clarity of the voice signal.

[63] Thereafter, the audio mixer may generate a single audio output signal by selectively mixing the identified voice channel signals. In this process, the audio mixer may adjust the volume balance between the channels so that each actor's voice is properly transmitted. In addition, the stereo image (e.g., left / right distribution) of the sound signal may be optimized through pan adjustment, and the clarity of the voice signal may be maintained by using an equalizer. In addition, in order to route the mixed voice signal to the output bus, a path may be set in the routing table of the mixer so that the voice signal can be transmitted to the output bus. In an exemplary embodiment, the routed signal may be converted into a digital signal as needed and transmitted in a form suitable for further processing or transmission. Thereafter, the converted signal may be transmitted to an audio interface or other output devices and be provided to a playback system or subtitle generation system within the performance venue.

[64] In an exemplary embodiment, important voice information such as an actor's dialog may be accurately transmitted without latency by processing signals in real time during the performance. To this end, the audio mixer may minimize latency by utilizing a high-performance digital signal processing (DSP) engine.

[65] In addition, the processor 130 may receive the chunk data, where audio is segmented into chunks, from the desktop application, may accumulate the received audio chunks for a certain period of time, and may input the accumulated audio data to the STT model to convert into a text. The processor 130 may receive audio chunk data transmitted from the desktop application through a network (local or cloud-based). FIG. 5 is a view for illustrating a function of a desktop application according to an exemplary embodiment. In an exemplary embodiment, the audio chunk may be digital audio data segmented in a unit of certain time, and each chunk may be configured by a fixed size and a sampling rate. In an exemplary embodiment, in order to prevent the loss of the chunk data, the processor 130 may verify the integrity of the data by applying an error verification technology (e.g., CRC, Checksum) when receiving the data. In addition, the received chunk data may be temporarily stored in a buffer within the processor 130. In an exemplary embodiment, the buffer may be used to accumulate a certain amount of audio data by accumulating chunk data. In an exemplary embodiment, the accumulated audio data may be processed to the next step when the minimum input length required by the STT model is reached or a specified time (e.g., 1 to 2 seconds) elapses. Thereafter, the processor 130 may apply an algorithm for aligning the data boundaries or maintaining continuity between audio samples in order for the processor to prevent a duplication or omission problem that occurs at the boundary of data segmented in a unit of chunks.

[66] The accumulated audio data may be converted into a sampling rate, a bit depth, and a data format (e.g., PCM, WAV) required by the STT model. In an exemplary embodiment, the data may be converted into a form optimized to be inputted into the STT model through data decompression or normalization. Thereafter, the processor 130 may transmit the prepared audio data to the STT model in real time, and the model may analyze the inputted audio data to convert into a text. The processor 130 may receive text data returned from the STT model. The result of the conversion may be recorded together with time information (timestamp) of the audio to support synchronization processing. In an exemplary embodiment, after converting into the text, the processor may compare the STT result with a predefined script to calculate a matching rate (matching based on similarity) and verify the final dialog. When the quality of the converted text is below a certain standard, additional processing (e.g., acoustical analysis or reprocessing) may be performed. The final text data may be transmitted to the desktop application, and be stored in a local database if necessary, to be utilized for subsequent subtitle generation, translation, or recording purposes.

[67] In addition, the processor 130 may proceed with the search for a certain number of script sentences (e.g., 5) after the previously matched index of the script, and may determine the detected dialog as a matching dialog when detecting a script exceeding a threshold value by measuring distance-based similarity. Hereinafter, the process of determining the matching dialog and displaying the script data will be described in more detail with reference to FIG. 6. FIG. 6 is a view showing a process of displaying the received script data in real time on a screen of a display in smart glasses according to an exemplary embodiment. In an exemplary embodiment, the processor 130 may record the index of the previously matched script and may set a reference point to start a new search. The reference point set in the exemplary embodiment may indicate the location of a recently identified dialog in the script data. In addition, the processor 130 may minimize the amount of computation by proceeding with the search only for a certain number (e.g., five sentences) after the previously matched index. The search range may be dynamically adjusted in consideration of the performance progress speed and the script structure. Thereafter, the processor 130 may receive as an input the text data generated through the Speech-to-Text (STT) model and may preprocess the same into an analyzable form. In an exemplary embodiment, the preprocessing process may remove unnecessary space and special characters, keep uppercase or lowercase unified, normalize to handle pronunciation and grammatical variations, and call script sentence data. In addition, the processor 130 may sequentially call script sentences within the search range and generate a list of comparison target. Thereafter, the processor 130 may calculate a distance-based similarity between the STT conversion text and the searched script sentence. For the similarity calculation, methods such as an edit distance calculation, a cosine similarity calculation, a Jaccard similarity calculation, or a Levenshtein distancebased similarity calculation, and the like may be used, but are not limited thereto. The edit distance (Levenshtein Distance) method may measure the difference by calculating the number of insertion, deletion, and replacement operations between two texts, and the cosine similarity may vectorize the text and then measure the similarity on the basis of the angle between each vector. Jaccard Similarity calculation may calculate the ratio of intersections and unions between sets of words in two texts. In the Levenshtein Distance similarity calculation, Levenshtein distance may be a type of edit distance for calculating the similarity between two strings. This distance may represent the minimum number of editing operations (insertion, deletion, and replacement) required to convert one string into another string.

[68] In an exemplary embodiment, the processor 130 may set a threshold value in advance for determining the similarity, and for example, when a similarity of 0.8 (80%) or higher is satisfied, two texts may be considered to be matched. In an exemplary embodiment, the threshold value may be adjusted according to a performance environment. Thereafter, the processor 130 may select a sentence with the highest similarity value among the searched script sentences, and when exceeding the threshold value, the selected sentence may be detected as a matching dialog. When there is no sentence exceeding the threshold value, the processor 130 may compare the additional sentence by expanding the search range. In addition, the STT converted text can be reprocessed (noise removal, additional filtering, etc.) and the user may be asked to check the manual.

[69] Thereafter, the processor 130 may store the index of the detected matching dialog and utilize the same as the reference point for subsequent search, and the matched dialog may be stored together with the timestamp to be used as a synchronized subtitle or record. In an exemplary embodiment, the final matching dialog may be transmitted to the desktop application and then may be displayed in real time on the AR subtitle device or the display device.

[70] In addition, the processor 130 may optimize the computational speed by limiting the script sentences of the search range to fewer than a certain number. The processor 130 may record the index of the previously matched script and set the same as the reference point for a new search. The reference point may reflect the synchronization state of the performance progress situation and the script. Thereafter, the processor 130 may set only a certain number (e.g., 5 or less) of script sentences after the reference point as search targets. This range may be set in advance in consideration of the structure of the script and the average interval between dialogs, or may be dynamically adjusted during the performance. In an exemplary embodiment, the processor 130 may load into the memory and process only the script sentence corresponding to the search range after the reference point. This may prevent memory overload that occurs when searching the entire script. In addition, script sentences outside the search range may be excluded from the computation target, so that unnecessary computation can be reduced. In order to satisfy the real-time demand of the performance, the processor 130 may update the search range in real time. In an exemplary embodiment, by limiting the search range, the processor may minimize the amount of data to be processed and shorten the time required for search and comparison operations. In addition, since the processor 130 calculates the distance-based similarity (e.g., Levenstein distance) only within the search range, there may be no need to repeatedly calculate the similarity for a large dataset. The reduction in the amount of computation may improve the reaction speed of the real-time subtitle generation system.

[71] In addition, the processor 130 may dynamically adjust the search range according to the situation during the performance. For example, when the interval between dialogs is short, the search range may be reduced, and when the interval is long, the search range may be expanded. Through this, the system may optimize the computation speed while maintaining the accuracy of dialog matching. In addition, in an exemplary embodiment, when a dialog whose similarity exceeds the threshold value is not found within the search range, the processor 130 may call an additional script sentence by expanding the search range and may proceed with a comparison. In an exemplary embodiment, when a matched dialog is identified within the searched range, the corresponding dialog may be recorded by the processor and transmitted to the desktop application. In addition, the index of the matched dialog may be set as a new reference point, and the next search may be prepared.

[72] In an exemplary embodiment, the processor 130 may transmit the matched script to the desktop application when the matching is successful, and may initialize the existing audio chunk. To this end, the processor 130 may check the result of calculating the distance-based similarity between the STT (Speech-to-Text) converted text and the script, and may determine that the matching is successful when exceeding the threshold value. In addition, the script sentence with the highest similarity may be selected as the matched script, and the corresponding information may be transmitted to the desktop application. In an exemplary embodiment, the matched script may be transmitted to the desktop application through an internal network (local network) or other communication protocols (e.g., WebSocket). The transmitted data may include the matched script text, timestamp information of the dialog, and metadata (e.g., an actor's name, a scene number, etc.) .

[73] In an exemplary embodiment, the processor 130 may transmit data in a lightweight format to minimize the latency in the transmission process. For example, the data may be configured in JSON format to reduce the data size and improve the processing speed. In addition, the processor 130 may delete or initialize the existing audio chunk data accumulated in the internal buffer after a successful match. To this end, the processor 130 may prepare for receiving the next audio data by emptying the buffer memory and may optimize the memory usage by preventing unnecessary data residue.

[74] In an exemplary embodiment, the initialization may be performed immediately after the script is transmitted, and space may be secured for a new audio chunk to be subsequently received. After the initialization, the processor 130 may complete the preparation to receive the next audio chunk data. In an exemplary embodiment, the newly received data may be accumulated again in the initialization state and on the basis of this, a new matching operation may be started. In an exemplary embodiment, a start point of the subsequent search and matching operation may be set by recording the index of the matched script as the reference point. In addition, when the matching fails or the similarity is below the threshold value, the processor 130 may not initialize the audio chunk and may store the same in the buffer. The processor 130 may accumulate additional data to retry the search, or may expand the search range to proceed with the next matching operation.

[75] In addition, the desktop application may transmit real-time script data to an output device including smart glasses by using a WebSocket protocol through the local network, and the smart glasses may display the received script data on the screen of the display in real time. The desktop application may operate as a WebSocket server, and the smart glasses or other output devices may be set as a WebSocket client. In an exemplary embodiment, a client may request a connection to a server on the basis of the local network (IP address and port number). In an exemplary embodiment, the WebSocket connection may be performed on top of the TCP protocol, and the client may generate a continuous two-way communication channel through a handshake process with the server. When successfully connected, both sides may be in a state where data can be exchanged in real time. Afterwards, the desktop application may convert the matched script data into JSON, XML, or other lightweight data formats. In addition, a transmission queue may be generated for managing real-time script data, thereby providing a mechanism of ensuring the order of data and preventing a loss. The script data prepared in an exemplary embodiment may be transmitted to the output device (smart glasses) through a WebSocket protocol. Data may be transmitted quickly and stably through two-way communication. In an exemplary embodiment, the data may be transmitted in UTF-8 encoded text or binary format. Text-based message format (JSON or simple text) may be mainly preferred. In addition, the desktop application may synchronize the transmission period of the script data with the performance progress speed. For example, the corresponding data may be transmitted to the output device together with a timestamp according to the dialog time. In an exemplary embodiment, the smart glasses or other output devices may operate as a client and may receive data transmitted through a WebSocket connection. Thereafter, the received data may be parsed and displayed on the screen. The script contents may be visually provided in real time through a head-up display (HUD) of the smart glasses or other display interfaces. Since WebSocket support two-way communication, the smart glasses can periodically notify the desktop application of the connection state. For example, the connection state can be checked by using ping / Pong messages, and reconnection may be automatically attempted when a network disconnection or error occurs. In addition, in an exemplary embodiment, in the event of data loss or disconnection, the desktop application may retransmit untransmitted data or may operate a complementary mechanism to recover missing data.

[76] When the performance ends, the desktop application may normally terminate the WebSocket connection with the smart glasses. To this end, a termination message may be exchanged between the client and the server.

[77] After the connection is terminated, the desktop application and output device may maintain the efficiency of the system by releasing the used memory and network resources.

[78] In addition, the processor 130 may attempt state transitions according to the threshold while receiving each chunk, in order to perform a state machine-based voice chunk analysis, and may determine whether to transmit or initialize dialogs after checking the degree of transcription similarity according to the current state. FIG. 7 is a view showing a voice chunk state search algorithm according to an exemplary embodiment. In an exemplary embodiment, the state machine may include the following main states by dividing the voice chunk analysis process step by step.

[79] idle state: initial state, waiting for analysis to start

[80] accumulating state: collecting and accumulating voice chunk data

[81] processing state: analysis and transcription of accumulated data

[82] matched state: transcription results are matched to the script

[83] error state: below threshold or matching failure

[84] In an exemplary embodiment, the processor 130 may start in the idle state when the system is initialized, and may transition to the accumulating state as receiving voice chunk data. Thereafter, the processor 130 may continuously receive voice chunk data transmitted from the STT system or desktop application. Thereafter, the processor 130 may transition from the accumulating state to the idle state, when the voice chunk data starts to be received, and may transition from the accumulating state to the processing state when the chunk accumulates for a certain period of time, or when the data size reaches the threshold value. In addition, the case, where the processing state is transitioned to the matched state, may be when the transcription results are matched to the script and the similarity exceeds the threshold value. In addition, when the matching fails, the processing state may be transitioned to the error state.

[85] In an exemplary embodiment, in the processing state, the processor 130 may input the accumulated voice chunk data into the Speech-to-Text (STT) model and may convert the same into the text. In addition, the similarity between the transcribed text and the sentences within the current search range of the script may be measured. A distance-based algorithm (e.g., Levenstein distance) may be used to measure the similarity, and when the similarity exceeds the threshold value (e.g., 80% or more), it may be considered as a matching. In an exemplary embodiment, when the transcription similarity exceeds the threshold value, it may be transitioned to the matched state, and the matched script text and timestamp data may be transmitted to the desktop application. Thereafter, the speech chunk data up to now may be initialized to prepare for receiving new data, and when the matching fails, it may be transitioned to the error state. When transitioned to the error state, the matching may be retried by preserving the existing chunk data, accumulating and re-analyzing additional data, expanding the search range, or correcting the quality of the accumulated data. Then, after the dialog is successfully transmitted, the processor may return to the idle state and wait for new voice chunk data. In an exemplary embodiment, when the chunk data is lost or incomplete, the processor 130 may maintain the current state and attempt to receive again. Thereafter, when a network error occurs during the transmission process, the processor may temporarily store the data locally and may resume the transmission when the connection is restored.

[86] A script extraction and subtitle transmission device synchronized with live performing art contents according to an exemplary embodiment can provide subtitles in a language preferred by the audience in performances such as plays, musicals, and operas performed in foreign languages, thereby removing language barriers.

[87] In addition, in an exemplary embodiment, it may be possible to foster an environment, where various audiences can understand the same performance regardless of what language they speak, and to help deaf or hard-of-hearing audiences to understand the dialog of the performance through real-time subtitles, thereby improving accessibility to performance viewing. In addition, by complementarily providing visual information in addition to voice-based information transmission, audiences can see the synchronized dialog in real time in front of their eyes, and can immediately understand the contents of the performance without missing anything.

[88] In addition, in an exemplary embodiment, the script and performance can be synchronized so that the context of the scene can be viewed without missing, and accurate subtitles can be provided in real time through STT AI and scenario mapping.

[89] In addition, in an exemplary embodiment, the accuracy of dialog can be improved by removing unnecessary background music and by utilizing only voice channels, and the reliability of subtitle provision can be improved by matching the optimal dialog through distance-based similarity measurement.

[90] In addition, through an exemplary embodiment, the usability of the audience can be maximized by utilizing display devices such as AR smart subtitle glasses, and real-time data transmission through WebSocket can enable fast and stable data transmission.

[91] In addition, through an exemplary embodiment, it may be possible to process in real time various information generated during the performance so that the actor's impromptu dialog or changed matters in the performance can be quickly reflected in subtitles, and technical latency during the performance may be minimized to maintain the viewing flow. In addition, a new type of experience may be provided for allowing the audience to watch subtitles simultaneously linked to the performance through a smart device.

[92] Furthermore, the original version of performing art contents can be viewed anywhere overseas without language barriers through AR subtitle glasses, and the global performing art industry can enter into a new market by providing a service that breaks down language barriers.

[93] In addition, through an exemplary embodiment, AI acoustic feature extraction and voice-subtitle conversion technology can be utilized to provide a subtitle transmission service synchronized with the contents during live performances of performing arts, and a service can be provided that allows audiences to set their preferred language to improve their understanding and immersion in the performance even with language barriers in the original work.

[94] In addition, through an exemplary embodiment, the audience can watch the performance through the worn AR subtitle glasses while simultaneously observing the actor's acting and movement, and have a more comfortable and enjoyable viewing experience. This can not only increase audience satisfaction, but also contribute to improving the popularity and reputation of the performance.

[95] In addition, through an exemplary embodiment, it can be possible to provide performances where performing art contents around the world is presented in localized language expressions .

[96] In addition, since performing arts are contents that attract many audiences of various nationalities and languages, it can be possible to provide a more convenient and comfortable viewing environment for multicultural audiences by introducing subtitle glasses technology. Multicultural audiences can see and understand subtitles translated into their native language, which can provide a more friendly and inclusive performance environment.

[97] In addition, through the subtitle glasses provided in an exemplary embodiment, the audience can watch the performance while simultaneously observing the actor's acting and movement, and have a more comfortable and enjoyable viewing experience. This can not only increase audience satisfaction, but also contribute to improving the popularity and reputation of the performance.

[98] Meanwhile, the methods according to various exemplary embodiments of the present disclosure described above may be implemented in the form of an application or a software program that can be installed in an existing electronic device.

[99] In addition, all or part of the method may be configured as several software function modules and implemented in an operating system (OS). Alternatively, each step may be configured as one software function module, or each step may be combined to be configured as one software function module and implemented on the operating system. Therefore, even when some exemplary embodiments of the present disclosure are not implemented entirely as one software function module, it may be understood that the method of the present disclosure is implemented when several software function modules implement each step of the present disclosure and several software function modules are implemented in one operating system.

[100] In addition, the methods according to the various exemplary embodiments of the present disclosure described above may be implemented only with a software upgrade or a hardware upgrade for an existing electronic device. In addition, the various exemplary embodiments of the present disclosure described above may be performed through an embedded server provided in the electronic device or an external server of the electronic device.

[101] Meanwhile, according to an exemplary embodiment of the present disclosure, the various exemplary embodiments described above may be implemented as software including commands stored in a computer-readable recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the exemplary embodiments described in the present specification may be implemented as the processor itself. By a software implementation, the exemplary embodiments, such as the procedures and functions described in this specification, may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described in the present specification.

[102] Meanwhile, a computer or similar device may be a device capable of calling a command stored in a storage medium and operating according to the called command, and may include a device according to the disclosed exemplary embodiments. When the command is executed by the processor, the processor may perform a function corresponding to the command directly or by using other components under the control of the processor. The command may include codes generated or executed by a compiler or an interpreter.

[103] The device-readable recording medium may be provided in the form of a non-transitory computer readable recording medium. Herein, "non-transitory" may mean that the storage medium does not contain signals and is tangible, but may not distinguish that data is stored in the storage medium semi-permanently or temporarily. In this case, the non-transitory computer-readable medium may be a medium that semi-permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, like a register, cache, memory, and the like. Specific examples of the non-transitory computer-readable medium may include CDs, DVDs, hard disks, Blu-ray disks, USB, a memory card, ROM, and the like.

[104] As described above, exemplary embodiments are disclosed in the drawings and in the specification. Although exemplary embodiments are described using specific terms in the present specification, this may be used only for the purpose of describing the technical idea of the present disclosure, and may not be used to limit the meaning or the scope of the present disclosure described in the claims. Therefore, those skilled in the art will understand that various modifications and other equivalent exemplary embodiments are possible therefrom. Therefore, the true technical protection scope of the present disclosure should be determined by the technical idea of the appended claims. Industrial Applicability

[105] A script extraction and subtitle transmission device synchronized with live performing art contents according to an exemplary embodiment can provide subtitles in a language preferred by the audience in performances such as plays, musicals, and operas performed in foreign languages, thereby removing language barriers.

[106] In addition, in an exemplary embodiment, it may be possible to foster an environment, where various audiences can understand the same performance regardless of what language they speak, and to help deaf or hard-of-hearing audiences to understand the dialog of the performance through real-time subtitles, thereby improving accessibility to performance viewing. In addition, by complementarily providing visual 5 information in addition to voice-based information transmission, audiences can see the synchronized dialog in real time in front of their eyes, and can immediately understand the contents of the performance without missing anything. 10

Claims

1. A script extraction and subtitle transmission system synchronized with live performing art contents, the system comprising:a microphone for collecting sound containing an actor's dialog and music during live performances including plays, musicals, and operas in a performing art field;a server for removing background music from the sound collected from the microphone and filtering only vocals to input into a Speech-to-Text (STT) model, for comparing dialog information converted into a text with an original transcript content to recognize a synchronization point, and for mapping with a scenario stored in advance to transmit subtitles onto an output device according to a mapping result;an audio mixer for receiving the sound as an input to adjust volume, tone, and panning of each signal, and for mixing and transmitting as a single output signal; andan output device for outputting the subtitles by projecting the subtitles received from the server onto a display, and for outputting the output signal.

2. The system of claim 1, wherein the server comprises:a memory for storing at least one command for a script extraction and subtitle transmission synchronized with the liveperforming art contents; anda processor for performing operations according to the command,wherein the processor captures voice from the sound transmitted from the microphone, and transmits the captured voice to an individual channel of the audio mixer andthe audio mixer mixes and routes voice channel signals to an output bus while excluding unnecessary background sounds or music channels.

3. The system of claim 2, wherein the processor receives chunk data, where audio is segmented in a unit of chunks, from a desktop application, accumulates the received audio chunks for a certain period of time, and inputs the accumulated audio data into the STT model to convert into a text.

4. The system of claim 3, wherein the processor proceeds with a search on a certain number of script sentences after a previously matched index of the script, and when a script exceeding a threshold value is detected by measuring distancebased similarity, the detected dialog is determined as a matching dialog.

5. The system of claim 4, wherein the processor limits the script sentences of a search range to fewer than a certain number.

6. The system of claim 4, wherein in the event of matching success, the processor transmits a matched script to the desktopapplication and initializes the existing audio chunk.

57. The system of claim 6, wherein the desktop application transmits real-time script data to the output device including smart glasses by using a WebSocket protocol through a local network, and10 the smart glasses displays the received script data on ascreen of a display in real time.

8. The system of claim 3, wherein the processor attempts state transitions according to a threshold while receiving each15 chunk in order to perform a state machine-based voice chunk analysis, and determines whether to transmit or initialize the dialog after checking transcript similarity according to the current state.s

Images