Live conference real-time subtitle generation method and system based on AI fusion

Abstract

The application provides a live conference real-time subtitle generation method and system fused with AI, relates to the field of real-time subtitle generation, and comprises the following steps: collecting an audio stream and dividing the audio stream into segments, assigning a sound source identifier to each audio subsegment after sound source separation and identification; performing speech recognition on the identified audio and associating the sound source to generate identified text; correcting text ambiguity by using a context knowledge base containing a subject word table and historical text; cutting the text according to an audio energy envelope to form a subtitle unit; calculating a subtitle display time, assigning a vertical offset to subtitles with overlapping time but different sound sources to realize spatial separation; adjusting the subtitle display coordinates in combination with a disabled area determined by a video frame foreground, and rendering and outputting. The method realizes clear distinction and accurate presentation of multi-speaker subtitles, and improves the readability and user experience of conference live subtitles.

Description

Technical Field

[0001] This invention relates to the field of real-time caption generation technology, and in particular to a method and system for generating real-time captions for live conferences that integrates AI. Background Technology

[0002] In live video conferencing scenarios, caption generation technology is crucial for improving information delivery efficiency and user experience. Existing solutions typically employ automatic speech recognition systems to transcribe the conference audio stream in real time and then overlay the generated text directly onto the video screen as captions.

[0003] Conventional approaches rely on a single speech recognition engine to process mixed audio streams as a whole. This method has significant limitations when dealing with complex meeting scenarios where multiple people speak simultaneously. The system struggles to distinguish the voices of different speakers, resulting in mixed transcribed text and a lack of clear speaker attribution information. Participants find it difficult to quickly identify which participant is speaking, reducing the readability and reference value of the captioned information.

[0004] Furthermore, existing caption generation systems employ relatively simple strategies for the spatiotemporal layout of text presentation. The system typically displays the recognized text line by line in a fixed area of ​​the screen (such as the bottom) in chronological order. When multiple people speak rapidly or briefly overlap during a meeting, the display time of the captions can easily overlap, causing text stacking and mutual obstruction. This severely interferes with the user's visual tracking and content comprehension, particularly impacting the viewing experience during information-intensive periods.

[0005] Another common shortcoming lies in the insufficient interaction between subtitles and video content. Conventional systems often render subtitles in preset static areas, failing to adequately consider key visual information present in the video frame. For example, when the speaker's image or important presentation charts are located at the bottom of the screen, subtitles fixed in that area will obscure these key visual elements, resulting in information loss. Existing technology lacks intelligent perception of the screen content and cannot dynamically adjust the subtitle position to avoid obscuring important foreground targets.

[0006] In summary, existing real-time conference caption generation technologies have shortcomings in distinguishing multiple speakers, optimizing overlapping caption layouts, and adaptively avoiding overlap between captions and screen content, which affects the clarity, readability, and overall viewing experience of caption information. Summary of the Invention

[0007] This invention provides a method and system for generating real-time subtitles for live conferences that integrates AI, which can solve the problems in the prior art.

[0008] A first aspect of this invention provides a method for generating real-time subtitles for live video conferencing that integrates AI, comprising:

[0009] Collect audio stream data and divide audio segments according to time windows. Separate and identify the sound sources of each audio segment and extract audio sub-segments. Assign sound source identifiers to each audio sub-segment and generate an identifiable audio set.

[0010] Speech recognition is performed on each audio sub-segment in the identified audio set, and sound source identifiers are associated with them to generate an identified text set;

[0011] Construct a contextual knowledge base containing a thesaurus of conference topics and historical texts. Based on the contextual knowledge base, match and correct ambiguous words in the identified text set to generate a corrected identified text set.

[0012] Extract the energy envelope of each audio sub-segment, determine the segmentation point based on the position of the energy envelope trough, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a subtitle unit set;

[0013] The display time is calculated based on the duration of the audio sub-segment corresponding to each subtitle unit in the subtitle unit set. Subtitle units with overlapping display times are identified and vertical offsets are allocated according to the differences in sound source identifiers to generate a spatial separation scheme.

[0014] Extract the foreground target area of ​​the video frame to determine the subtitle ban area, adjust the display coordinates of the subtitle unit according to the subtitle ban area and spatial separation scheme, and render it to the video frame to complete the subtitle output.

[0015] The audio stream data is acquired and divided into audio segments according to time windows. Sound sources are separated and identified within each audio segment, and audio sub-segments are extracted. Sound source identifiers are assigned to each audio sub-segment, generating an identifiable audio set including:

[0016] The system collects audio stream data and detects energy changes in the audio stream data. Based on the energy changes, it identifies silent intervals and active intervals. The silent intervals are used as the boundaries of audio segments, and the active intervals are divided into audio segments according to time windows.

[0017] The audio segment is subjected to short-time Fourier transform to obtain the time-frequency matrix. The harmonic structure and formant distribution are extracted from the time-frequency matrix to construct the sound source mask matrix. The time-frequency matrix and the sound source mask matrix are multiplied element by element to separate the time-frequency components of each sound source.

[0018] The audio sub-segments are reconstructed by performing inverse short-time Fourier transform on the time-frequency components of each sound source. The Mel-frequency cepstral coefficients of each audio sub-segment are extracted and combined with the fundamental frequency trajectory to form a voiceprint descriptor. The Mahalanobis distance between the voiceprint descriptors is calculated and spectral clustering is performed to obtain the sound source clustering results.

[0019] Assign a unique sound source identifier to the cluster center in the sound source clustering results, calculate the signal-to-noise ratio distribution of the audio sub-segment corresponding to each sound source identifier, assign a confidence weight to each sound source identifier according to the signal-to-noise ratio distribution, associate the sound source identifier with the confidence weight to the corresponding audio sub-segment, and generate an identifier audio set.

[0020] Speech recognition is performed on each audio sub-segment in the tagged audio set, and sound source identifiers are associated with them to generate a tagged text set, including:

[0021] Extract each audio sub-segment and its corresponding sound source identifier from the identified audio set. Perform phoneme segmentation on each audio sub-segment to obtain a phoneme sequence. Perform forward and backward traversal on the phoneme sequence to obtain forward and backward dependencies. Merge the forward and backward dependencies to obtain a bidirectional dependency.

[0022] Based on the bidirectional dependency relationship, the phoneme sequence is combined into multiple candidate texts, and the degree of matching between each candidate text and the phoneme sequence is calculated to obtain the acoustic score;

[0023] The language score is obtained by calculating the transition probabilities between words within each candidate text.

[0024] The acoustic score and the language score are weighted and summed to obtain a comprehensive score. The candidate text with the highest comprehensive score is selected as the recognition text.

[0025] The sound source identifiers of each recognized text and the corresponding audio sub-segment are associated. The number of characters in the recognized text corresponding to each sound source identifier and the duration of the audio sub-segment are counted. The speech rate value of each sound source identifier is calculated based on the number of characters and the duration.

[0026] Based on the speech rate value, a speech volume weight is assigned to each sound source identifier, and the identified text, sound source identifier, and speech volume weight are combined to generate an identifier text set.

[0027] A contextual knowledge base containing a thesaurus of conference topics and historical texts is constructed. Based on the contextual knowledge base, ambiguous words in the identified text set are matched and corrected, generating a corrected identified text set including:

[0028] Extract core keywords from the identified text set to construct a conference keyword list; extract speech texts from historical conference data as historical texts to construct a sound source semantic profile; identify topic switching points from the identified text set to extract core words from topic segments to construct topic segment context features.

[0029] A ternary association network is established by combining the conference theme thesaurus, the semantic profile of the sound source, and the contextual features of the theme segment. The ternary association network is then combined with historical texts to construct a contextual knowledge base that includes the conference theme thesaurus and historical texts.

[0030] The text is extracted from the identified text set to detect ambiguous words and extract local context. Based on the context knowledge base, the shortest path length of the local context and each semantic interpretation of the ambiguous word in the ternary association network is calculated to obtain the context tightness. The number of paths through each semantic interpretation is counted to obtain the semantic support. The context tightness and semantic support are combined to calculate the context dependency strength.

[0031] Based on the context knowledge base, ambiguous words in the identified text set are matched and corrected. The semantic interpretation with the strongest context dependence is selected to replace the ambiguous word to obtain the corrected text.

[0032] The corrected text is combined with the corresponding sound source identifier to generate a set of corrected identifier texts. The standard words in the corrected text are backpropagated to update the ternary association network to obtain an updated contextual knowledge base, which is used for the correction of ambiguous words in the identifier text set in the next time window.

[0033] Extract the energy envelope of each audio segment, determine the segmentation point based on the valley position of the energy envelope, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a set of subtitle units including:

[0034] Extract the energy envelope of each audio sub-segment to detect the valley position, and extract the corresponding text of each valley position in the correction label text set to construct a dependency relationship chain;

[0035] The ratio of the number to the length of dependency chains cut off at the trough position is used as the syntactic split degree. Trough positions with a syntactic split degree lower than a preset split threshold are selected, and the split point is determined based on the energy envelope trough position.

[0036] Historical speech texts for each sound source identifier are extracted from historical conference data to construct a sound source feature word list and word sequence pattern. The feature word coverage and word sequence matching degree of the window texts before and after each segmentation point in the corrected identifier text set are calculated under each sound source identifier.

[0037] The difference between coverage and matching degree of the same sound source identifier between adjacent segmentation points is calculated as the cross-segmentation point change amplitude, and the sound source identifier with the largest cross-segmentation point change amplitude is selected and associated with the segmentation point;

[0038] Based on the segmentation points, the corresponding text in the set of correction identifiers is segmented to obtain text fragments, and the coverage and matching degree of the text fragments under the associated sound source identifiers are calculated.

[0039] Text segments with coverage and matching accuracy below average are reassociated with the sound source identifiers with the highest coverage and matching accuracy. The sound source identifiers are kept associated, and the text segments are combined with the sound source identifiers and timestamps to generate a set of subtitle units.

[0040] The display time is calculated based on the duration of the audio sub-segment corresponding to each subtitle unit in the subtitle unit set. Subtitle units with overlapping display times are identified, and vertical offsets are allocated according to differences in sound source identifiers. A spatial separation scheme is generated, including:

[0041] Extract the duration and start timestamp of the audio sub-segment corresponding to each subtitle unit from the subtitle unit set to calculate the display time;

[0042] Extract role labels and speaking weights of each sound source identifier from historical conference data, calculate the speaking weight difference of each sound source identifier in the display time overlapping subtitle unit pair, and identify the main speaking subtitle unit and the secondary speaking subtitle unit in the display time overlapping subtitle unit pair based on the speaking weight difference.

[0043] Detect whether the start timestamp of the secondary speech caption unit is later than the start timestamp of the main speech caption unit, and mark the secondary speech caption unit whose start timestamp is later than the main speech caption unit as an interjection caption unit;

[0044] A spatial separation scheme is generated by assigning a center vertical offset to the main speech caption unit, an edge vertical offset to the interstitial speech caption unit, and a symmetrical vertical offset to the non-interstitial secondary speech caption unit.

[0045] Extracting the foreground target area of ​​the video frame to determine the subtitle-disallowed area, adjusting the display coordinates of the subtitle unit according to the subtitle-disallowed area and spatial separation scheme, and rendering it to the video frame to complete the subtitle output includes:

[0046] Semantic segmentation is performed on video frames to extract foreground target regions. The inter-frame displacement vectors and gradient consistency of pixels within the foreground target regions are calculated to construct a motion texture fusion matrix. Morphological dilation is performed on the motion texture fusion matrix to determine the subtitle-prohibited regions.

[0047] The vertical offset and sound source identifier of the subtitle unit are extracted from the spatial separation scheme. The display coordinates of the sound source identifier in the historical video frames are extracted to construct the sound source position probability distribution map. The initial display coordinates are calculated based on the probability peak and vertical offset of the sound source position probability distribution map.

[0048] Calculate the percentage of the intersection area between the initial display coordinates and the subtitle disabled area, filter subtitle units whose intersection area percentage exceeds the threshold as conflicting subtitle units, and sort them according to the weight of the number of words spoken;

[0049] Search for candidate display coordinates outside the subtitle-disabled area. Calculate the ratio of the probability density of the candidate display coordinates in the probability distribution map of the sound source location to the distance to the boundary of the subtitle-disabled area as a position stability index. Select the candidate display coordinates with the largest position stability index to replace the initial display coordinates to obtain the adjusted display coordinates.

[0050] The subtitle unit is rendered to the video frame based on the adjusted display coordinates to complete the subtitle output. The adjusted display coordinates are then accumulated to the probability distribution map of the corresponding sound source location.

[0051] A second aspect of this invention provides a real-time caption generation system for live video conferencing that integrates AI, comprising:

[0052] The audio processing unit is used to collect audio stream data and divide audio segments according to time windows, separate and identify each sound source in the audio segments and extract audio sub-segments, assign sound source identifiers to each audio sub-segment, and generate an identifier audio set.

[0053] The speech recognition unit is used to perform speech recognition on each audio sub-segment in the identified audio set and associate it with the sound source identifier to generate an identified text set.

[0054] The text correction unit is used to construct a contextual knowledge base containing a thesaurus of conference topics and historical texts. Based on the contextual knowledge base, it matches and corrects ambiguous words in the identified text set to generate a corrected identified text set.

[0055] The text segmentation unit is used to extract the energy envelope of each audio sub-segment, determine the segmentation point based on the position of the energy envelope trough, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a set of subtitle units.

[0056] The spatial allocation unit is used to calculate the display time based on the duration of the audio sub-segments corresponding to each subtitle unit in the subtitle unit set, identify subtitle units with overlapping display times, allocate vertical offsets based on differences in sound source identifiers, and generate a spatial separation scheme.

[0057] The subtitle rendering unit is used to extract the foreground target area of ​​the video frame to determine the subtitle disabled area, adjust the display coordinates of the subtitle unit according to the subtitle disabled area and spatial separation scheme, and render it to the video frame to complete the subtitle output.

[0058] A third aspect of the present invention provides an electronic device, comprising:

[0059] processor;

[0060] Memory used to store processor-executable instructions;

[0061] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0062] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0063] In this embodiment, by separating sound sources and assigning identifiers, different speakers can be accurately distinguished and tracked, effectively solving the problem of subtitle confusion when multiple people speak simultaneously. Combining contextual knowledge base for ambiguity correction significantly reduces the speech recognition error rate, ensuring the accuracy of technical terms and conference-specific vocabulary. Text segmentation based on audio energy envelope makes subtitle segmentation more consistent with natural language pauses, improving readability. Regarding subtitle layout, by calculating display time and identifying overlapping conflicts, vertical offsets can be automatically assigned to subtitles from different sound sources, achieving clear spatial separation of multiple subtitles. This avoids the visual confusion caused by traditional subtitle overlay, allowing viewers to easily distinguish and track multiple concurrent statements. Simultaneously, foreground targets in video frames are extracted to determine prohibited subtitle areas, intelligently avoiding key areas such as speakers and presentation content, ensuring that subtitle display does not obscure important visual information. By dynamically adjusting the display coordinates of subtitle units and rendering, organic integration of subtitles and video footage is achieved. The entire process achieves fully automated processing from audio acquisition to subtitle rendering, providing highly accurate, highly readable, and logically laid-out real-time subtitles while maintaining low latency. This method is particularly suitable for online meetings and live broadcasts with multiple speakers, high level of expertise, and rich visual information. It greatly improves the information access experience for hearing-impaired individuals and cross-language participants, and enhances the overall efficiency and inclusivity of meeting communication. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating the AI-integrated real-time subtitle generation method for live video conferencing, as described in an embodiment of the present invention.

[0065] Figure 2 This is a flowchart illustrating the audio segmentation and sound source identification association process according to an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0068] Figure 1 This is a flowchart illustrating the AI-integrated real-time caption generation method for live video conferencing, as described in an embodiment of the present invention. Figure 1As shown, the method includes:

[0069] Collect audio stream data and divide audio segments according to time windows. Separate and identify the sound sources of each audio segment and extract audio sub-segments. Assign sound source identifiers to each audio sub-segment and generate an identifiable audio set.

[0070] Speech recognition is performed on each audio sub-segment in the identified audio set, and sound source identifiers are associated with them to generate an identified text set;

[0071] Construct a contextual knowledge base containing a thesaurus of conference topics and historical texts. Based on the contextual knowledge base, match and correct ambiguous words in the identified text set to generate a corrected identified text set.

[0072] Extract the energy envelope of each audio sub-segment, determine the segmentation point based on the position of the energy envelope trough, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a subtitle unit set;

[0073] The display time is calculated based on the duration of the audio sub-segment corresponding to each subtitle unit in the subtitle unit set. Subtitle units with overlapping display times are identified and vertical offsets are allocated according to the differences in sound source identifiers to generate a spatial separation scheme.

[0074] Extract the foreground target area of ​​the video frame to determine the subtitle ban area, adjust the display coordinates of the subtitle unit according to the subtitle ban area and spatial separation scheme, and render it to the video frame to complete the subtitle output.

[0075] The audio stream data is acquired and divided into audio segments according to time windows. Sound sources are separated and identified within each audio segment, and audio sub-segments are extracted. Sound source identifiers are assigned to each audio sub-segment, generating an identifiable audio set including:

[0076] The system collects audio stream data and detects energy changes in the audio stream data. Based on the energy changes, it identifies silent intervals and active intervals. The silent intervals are used as the boundaries of audio segments, and the active intervals are divided into audio segments according to time windows.

[0077] The audio segment is subjected to short-time Fourier transform to obtain the time-frequency matrix. The harmonic structure and formant distribution are extracted from the time-frequency matrix to construct the sound source mask matrix. The time-frequency matrix and the sound source mask matrix are multiplied element by element to separate the time-frequency components of each sound source.

[0078] The audio sub-segments are reconstructed by performing inverse short-time Fourier transform on the time-frequency components of each sound source. The Mel-frequency cepstral coefficients of each audio sub-segment are extracted and combined with the fundamental frequency trajectory to form a voiceprint descriptor. The Mahalanobis distance between the voiceprint descriptors is calculated and spectral clustering is performed to obtain the sound source clustering results.

[0079] Assign a unique sound source identifier to the cluster center in the sound source clustering results, calculate the signal-to-noise ratio distribution of the audio sub-segment corresponding to each sound source identifier, assign a confidence weight to each sound source identifier according to the signal-to-noise ratio distribution, associate the sound source identifier with the confidence weight to the corresponding audio sub-segment, and generate an identifier audio set.

[0080] During the audio acquisition phase, mixed audio stream data from the live conference is acquired in real time using audio acquisition equipment. The sampling rate is set to 16kHz to ensure the spectral integrity of the speech signal. The short-time energy value of the audio stream is continuously monitored during acquisition, specifically quantified by calculating the sum of the squares of the amplitudes of all sampled points within each 20ms audio frame. A dynamic energy threshold is set. This threshold is determined by multiplying the median energy of the audio over the most recent 5 seconds by an adjustment factor of 1.2. When the energy of audio frames is consistently below a certain level for 150ms... The system identifies silent intervals as quiet areas and continuous regions with energy exceeding a threshold as active intervals. Using the start and end times of silent intervals as natural boundaries, a sliding window mechanism is employed to divide the active intervals. The window length is set to 2 seconds, with an overlap rate of 25%, ensuring smooth transitions between adjacent audio segments. This energy-based segmentation method avoids forced segmentation in the middle of a speech while ensuring that each audio segment contains relatively complete semantic units.

[0081] Perform a short-time Fourier transform on the segmented audio data using a Hamming window function with a window length of 512 points and a frame shift of 256 points to obtain the time-frequency matrix in the complex domain. ,in Indicates the time frame index. This represents the frequency index. When identifying harmonic structures in the time-frequency matrix, energy peak points that satisfy integer multiple relationships are searched along the frequency axis, specifically detecting the fundamental frequency. and its second to fifth harmonic components to The energy distribution. For human voice signals, the focus is on extracting formant features. By finding local energy maxima in the 0.3kHz to 3.5kHz frequency band, the first three formants are identified. , , Frequency location. Construct a sound source mask matrix. At that time, for those belonging to the first The time-frequency units of each sound source are assigned a value of 1, while other positions are assigned a value of 0. The mask matrix is ​​determined through iterative optimization based on the continuity of harmonic energy and the stability of resonant peaks. Element-wise multiplication is then performed between the time-frequency matrix and the corresponding sound source mask matrix. This allows for the independent time-frequency components of each sound source to be obtained, effectively achieving frequency domain separation of mixed speech.

[0082] For the separated time-frequency components of each sound source An inverse short-time Fourier transform (ISFT) is performed, and the time-domain audio signal is reconstructed using the overlap-addition method to generate corresponding audio sub-segments. To characterize the voiceprint features of each audio sub-segment, it is first pre-emphasized and filtered with a filter coefficient of 0.97 to enhance high-frequency components. Then, 13-dimensional Mel-frequency cepstral coefficients are extracted, with a frame length of 25ms and a frame shift of 10ms, obtained by performing a discrete cosine transform on the logarithmic energy output of the Mel filter bank. Simultaneously, the autocorrelation method is used to extract the fundamental frequency trajectory, with the detection range limited to 80Hz to 400Hz to cover the fundamental frequency range of human voice. The autocorrelation function is calculated for each frame of audio, and the delay time corresponding to the first peak is searched in the normalized autocorrelation function; its reciprocal is the fundamental frequency of that frame. The mean vector of the Mel-frequency cepstral coefficient sequence is combined with the median and standard deviation of the fundamental frequency trajectory to form a 15-dimensional voiceprint descriptor. Calculate any two voiceprint descriptors and Mahalanobis distance ,in The covariance matrix for all voiceprint descriptors is used, and Mahalanobis distance can eliminate the influence of correlation between feature dimensions. A similarity matrix is ​​constructed based on the calculated distance matrix, and a spectral clustering algorithm is used to classify audio segments into different sound source categories. The number of clusters is automatically determined by the silhouette coefficient evaluation, and in typical conference scenarios, the number of clusters is set between 2 and 8.

[0083] For each sound source category obtained from spectral clustering, the centroids of all speaker descriptors belonging to that category are calculated as cluster centers. A globally unique sound source identifier is assigned to each cluster center, and this identifier is marked using an incrementing integer encoding method. , To evaluate the quality of each audio segment, its signal-to-noise ratio (SNR) is calculated. Specifically, this is done by estimating the ratio of the average energy of the speech segment to the noise energy of the non-speech segment within the audio segment, expressed as: The unit is decibels. Statistically calculate the signal-to-noise ratio (SNR) of all audio segments under the same sound source identifier, and then calculate the mean of their distribution. with standard deviation Assign confidence weights to the sound source identifier based on the signal-to-noise ratio distribution. The calculation method involves normalizing the average signal-to-noise ratio (SNR) of the sound source to the range of 0 to 1. For audio sub-segments with an SNR higher than 20 dB, the confidence weight is set to 0.95 or higher; for those with an SNR lower than 10 dB, it is set to 0.6 or lower. Linear interpolation is used in the intermediate region. The sound source identifier, confidence weight, and corresponding timestamp information are associated with each audio sub-segment, using triples... Stored in the form of, where The start time of each audio segment in the original audio stream is recorded. After the above processing, all audio segments containing source identifiers and confidence information constitute an identified audio set. This set provides a clear basis for source differentiation in subsequent speech recognition, enabling the transcribed text to be accurately associated with different speakers, and allowing for reliability assessment of the recognition results based on confidence weights. The entire source separation and identification process is executed in real time under a streaming processing architecture, with the processing latency of a single 2-second audio segment controlled within 300ms, meeting the real-time requirements of live streaming scenarios.

[0084] Speech recognition is performed on each audio sub-segment in the tagged audio set, and sound source identifiers are associated with them to generate a tagged text set, including:

[0085] Extract each audio sub-segment and its corresponding sound source identifier from the identified audio set. Perform phoneme segmentation on each audio sub-segment to obtain a phoneme sequence. Perform forward and backward traversal on the phoneme sequence to obtain forward and backward dependencies. Merge the forward and backward dependencies to obtain a bidirectional dependency.

[0086] Based on the bidirectional dependency relationship, the phoneme sequence is combined into multiple candidate texts, and the degree of matching between each candidate text and the phoneme sequence is calculated to obtain the acoustic score;

[0087] The language score is obtained by calculating the transition probabilities between words within each candidate text.

[0088] The acoustic score and the language score are weighted and summed to obtain a comprehensive score. The candidate text with the highest comprehensive score is selected as the recognition text.

[0089] The sound source identifiers of each recognized text and the corresponding audio sub-segment are associated. The number of characters in the recognized text corresponding to each sound source identifier and the duration of the audio sub-segment are counted. The speech rate value of each sound source identifier is calculated based on the number of characters and the duration.

[0090] Based on the speech rate value, a speech volume weight is assigned to each sound source identifier, and the identified text, sound source identifier, and speech volume weight are combined to generate an identifier text set.

[0091] The audio segments and their associated source identifiers are extracted one by one from the identified audio set, obtaining the audio segments corresponding to source 1, source 2, and so on up to source M. For each audio segment, pre-emphasis processing is first performed to enhance high-frequency components. A windowing and framing operation is then used to convert the continuous audio signal into several frames, with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. Mel-frequency cepstral coefficients are extracted from each frame of audio data, and the feature vectors are input into an acoustic model for phoneme segmentation. The acoustic model employs a deep neural network structure. By analyzing the energy changes and spectral distribution characteristics of the feature vectors, the phoneme boundary positions are marked in the audio segments, decomposing the audio segments into basic articulatory units such as / a / , / i / , and / u / , forming a temporally arranged phoneme sequence.

[0092] For the acquired phoneme sequence, a forward traversal operation is performed, scanning sequentially from the beginning of the phoneme sequence. An encoder is constructed to encode all phonemes before the current position, capturing the influence of each phoneme on its subsequent phonemes, forming a forward dependency matrix. This matrix records the forward context information between phonemes, reflecting the influence of preceding phonemes on the pronunciation of subsequent phonemes during the pronunciation process. Subsequently, a backward traversal operation is performed, scanning sequentially from the end of the phoneme sequence backward. A decoder is constructed to encode all phonemes after the current position, capturing the influence of each phoneme on its preceding phonemes, forming a backward dependency matrix. This matrix records the backward context information between phonemes, reflecting the constraint effect of subsequent phonemes on the recognition accuracy of preceding phonemes. The forward and backward dependency matrices are concatenated and fused along the feature dimension. The feature vectors at corresponding positions undergo linear transformation and nonlinear activation through a fully connected layer to obtain bidirectional dependency features that comprehensively express the bidirectional context between phonemes.

[0093] Based on the bidirectional dependency characteristics, a beam search algorithm is used for path searching, combining phoneme sequences into multiple possible candidate texts. During the beam search process, the beam width is set to 16, and the 16 highest-scoring paths are retained at each time step for further expansion. The search process starts from the initial marker, sequentially adding the corresponding text symbols for each phoneme. When a phoneme combination corresponds to multiple texts, a branch path is generated, ultimately producing candidate texts 1 to K. For each candidate text, its matching degree with the phoneme sequence is calculated to obtain an acoustic score. The acoustic score is calculated as follows: the candidate text is converted back into a phoneme sequence, the feature vector of the converted phoneme sequence is extracted, and the similarity is calculated with the feature vector of the phoneme sequence extracted from the original audio sub-segment. The cosine similarity metric is used; a higher similarity value indicates a higher degree of matching between the candidate text and the actual pronunciation, and this similarity value is the acoustic score.

[0094] For each candidate text, analyze the transition probabilities between its internal words to calculate the language score. Segment the candidate text into words. For the candidate text "The weather is nice today", it is segmented into four words: "today", "weather", "is", and "nice". Query the transition probabilities of adjacent words from a pre-trained 10-gram language model, and calculate the probability of "today" transitioning to "weather", the probability of "weather" transitioning to "is", and the probability of "is" transitioning to "nice". After taking the logarithm of each transition probability and accumulating them, obtain the logarithmic probability value of the language model for the candidate text. This value reflects the degree to which the candidate text conforms to the natural language expression habit and is used as the language score. The higher the language score value, the more the word collocations of the candidate text conform to the grammar rules and expression habits.

[0095] Perform a weighted sum of the acoustic scores and language scores corresponding to each candidate text to calculate the comprehensive score. The weighted sum formula is , where represents the acoustic score, represents the language score, and are weight coefficients and satisfy . Adjust the weight coefficients according to the actual application scenario. For a noisy environment, appropriately increase the weight of the language score to 0.6, and for a quiet environment, increase the weight of the acoustic score to 0.7. Compare the comprehensive score values of all candidate texts, and select the candidate text with the highest comprehensive score as the recognized text for this audio sub-segment.

[0096] Associate and bind the obtained recognized text with the sound source identifier corresponding to its source audio sub-segment. For all audio sub-segments with the sound source identifier "Sound Source 1", count the total number of characters in their recognized texts, and at the same time accumulate the durations of these audio sub-segments. Calculate the speech rate value of Sound Source 1 by dividing the number of characters by the duration, with the unit of characters per second. Repeat the same statistical calculation operations for Sound Sources 2 to M to obtain the speech rate values corresponding to each sound source identifier. The speech rate value has the calculation formula , where represents the number of characters, represents the duration.

[0097] Based on the calculated speech rate values of each sound source identifier, analyze the speaking activity levels of different sound sources in the meeting. Take the proportion of the total number of characters corresponding to each sound source identifier to the total number of characters of all sound sources as the speaking volume weight. The speaking volume weight of the sound source identifier has the calculation formula , where represents the sound source corresponding number of characters, This represents the total number of sound sources. The speaking volume weight ranges from 0 to 1, with a higher value indicating that the sound source spoke more in the meeting. Each identified text segment, its corresponding sound source identifier, and the speaking volume weight of that sound source identifier are bound together as a triple to form a structured data entry. The structured data entries corresponding to all audio sub-segments are summarized to generate an identifier text set. This set fully records the speaking content, identity, and speaking activity information of each sound source, providing a data foundation for subsequent subtitle generation and display.

[0098] A contextual knowledge base containing a thesaurus of conference topics and historical texts is constructed. Based on the contextual knowledge base, ambiguous words in the identified text set are matched and corrected, generating a corrected identified text set including:

[0099] Extract core keywords from the identified text set to construct a conference keyword list; extract speech texts from historical conference data as historical texts to construct a sound source semantic profile; identify topic switching points from the identified text set to extract core words from topic segments to construct topic segment context features.

[0100] A ternary association network is established by combining the conference theme thesaurus, the semantic profile of the sound source, and the contextual features of the theme segment. The ternary association network is then combined with historical texts to construct a contextual knowledge base that includes the conference theme thesaurus and historical texts.

[0101] The text is extracted from the identified text set to detect ambiguous words and extract local context. Based on the context knowledge base, the shortest path length of the local context and each semantic interpretation of the ambiguous word in the ternary association network is calculated to obtain the context tightness. The number of paths through each semantic interpretation is counted to obtain the semantic support. The context tightness and semantic support are combined to calculate the context dependency strength.

[0102] Based on the context knowledge base, ambiguous words in the identified text set are matched and corrected. The semantic interpretation with the strongest context dependence is selected to replace the ambiguous word to obtain the corrected text.

[0103] The corrected text is combined with the corresponding sound source identifier to generate a set of corrected identifier texts. The standard words in the corrected text are backpropagated to update the ternary association network to obtain an updated contextual knowledge base, which is used for the correction of ambiguous words in the identifier text set in the next time window.

[0104] The set of identifiable audio segments obtained after sound source separation is converted into a set of identifiable text by a speech recognition engine. This set of identifiable text contains the recognized text corresponding to multiple sound sources, with each segment carrying a unique sound source identifier. Due to the existence of homophones, colloquial expressions, and misrecognition of technical terms during real-time speech recognition, a contextual knowledge base is needed to correct ambiguous words in the recognized text.

[0105] For each segment of identified text in the identified text set, the TF-IDF algorithm is used to extract the top 20 words with the highest term frequency-inverse document frequency (IF-IDF) values ​​as candidate core words. These candidate core words are then tagged with parts of speech, retaining nouns, verbs, and adjectives while removing pronouns, conjunctions, and other functional words. The selected words are then compared with the pre-set keywords for the conference theme using cosine similarity calculations. Words with a similarity greater than 0.6 are included in the conference theme thesaurus. This thesaurus is dynamically updated; the term frequency weights are recalculated every five time windows of audio segments to ensure that the thesaurus reflects the current conference discussion focus.

[0106] Retrieve historical meeting records related to the current meeting topic from the historical meeting database and extract the speech texts. Group the speech texts according to the source identifiers corresponding to the source texts, and calculate the word frequency distribution for all historical texts corresponding to each source identifier. For each source, identify proper nouns, idioms, and expressions that appear more than 10 times in its historical texts to form a feature word set for that source. Combine the feature word set with the source's speech topic in the current meeting to construct a semantic profile of the source. The semantic profile includes the source's professional domain preferences, expression habits, and commonly used terminology to help identify the source's tendency to use ambiguous words in its speech.

[0107] Topic switching points are detected on the time series of the identified text set. The lexical overlap rate of the identified text within two adjacent time windows is calculated. A topic switch is determined when the overlap rate is below 0.3 for three consecutive time windows. The identified text set is divided into multiple topic segments using the topic switching point as the boundary. For each topic segment, the five most frequent nouns and three most frequent verbs are extracted as the core words. Word vectors are used to represent the core words, and the semantic distance between core words is calculated. A connection edge is established when the cosine distance between any two core words is less than 0.4, forming a topic segment contextual feature map.

[0108] A ternary association network is constructed by using words from the conference thesaurus as the first type of nodes, feature words from the semantic profiles of each sound source as the second type of nodes, and core words from each topic segment as the third type of nodes. In this network, an edge is established when two words co-occur within the same topic segment, with the edge weight set to the normalized co-occurrence frequency value. When words from the conference thesaurus and feature words from a sound source's semantic profile co-occur more than five times in historical text, a cross-layer connection is established between the corresponding nodes. Semantic association edges are established when the semantic similarity between the core words of a topic segment and words from the conference thesaurus is greater than 0.7. An index mapping relationship is established between this ternary association network and a historical text database. Historical text serves as an external knowledge source to support semantic reasoning within the network, together forming a contextual knowledge base.

[0109] Each segment of identified text in the identified text set is traversed, and ambiguous words are detected using a combination of rule-based and statistical methods. The identified text is matched against a standard dictionary, and words with multiple pronunciations corresponding to different meanings are marked as ambiguous. For each detected ambiguous word, the five words before and after it are extracted as local context. The words in the local context are mapped to a ternary association network, and the nodes corresponding to these words are found.

[0110] For each possible semantic interpretation of an ambiguous word, a corresponding candidate node is searched in a ternary association network. Dijkstra's shortest path algorithm is used to calculate the shortest path length from each lexical node in the local context to the candidate semantic node. A weighted average of all path lengths is taken, with the weights determined by the distance decay between the context words and the ambiguous word; context words closer to the ambiguous word have higher weights. The reciprocal of the weighted average path length is used as the context tightness; a higher value indicates a stronger correlation between the semantic interpretation and the current context.

[0111] In the ternary association network, starting from local contextual word nodes and ending at semantic explanation nodes for ambiguous words, a breadth-first search algorithm is used to traverse all possible paths. The total number of paths passing through each semantic explanation node is counted. The number of paths reflects the support strength of the semantic explanation in the current context; a higher number indicates that the semantic explanation has established associations with more contextual elements in the network. The semantic support score is obtained by normalizing the number of paths, with a value ranging from 0 to 1.

[0112] Contextual dependency strength is calculated by combining contextual tightness and semantic support. A weighted summation method is used, with contextual tightness accounting for 0.6 and semantic support accounting for 0.4. This weighting is based on experimental statistical results, prioritizing the direct correlation between ambiguous words and their context while also considering the broad semantic support within the overall knowledge network. For each ambiguous word detected in the identified text set, the contextual dependency strength of all its possible semantic interpretations is calculated, and the interpretation with the highest value is selected as the correction result.

[0113] The ambiguous words in the identified text are replaced with standard words corresponding to the selected semantic interpretations to obtain the corrected text. The sound source identifier association between the corrected text and the original identified text remains unchanged. After traversing all identified texts in the identifier text set to complete the ambiguity correction, each corrected text is recombined with its corresponding sound source identifier to generate a corrected identifier text set. Each element in this set contains the corrected text content and its corresponding sound source identifier.

[0114] The corrected standard words are extracted from the proofread text; these words represent the correct expressions verified in the current context. These standard words are then mapped back to a ternary association network to enhance the edge weights between related nodes. Specifically, when a node corresponding to a standard word has a connection to a context word node, the weight of that edge is increased by 0.1; when there is no connection between a standard word and the core word node of the current topic segment, a new edge is created and its initial weight is set to 0.5. Through this backpropagation mechanism, the semantic associations verified in actual use are integrated into the knowledge base, making the topology and edge weight distribution of the ternary association network more consistent with the contextual characteristics of the current meeting.

[0115] The updated contextual knowledge base is used to process the set of tagged texts corresponding to audio segments in the next time window. As the conference progresses, the contextual knowledge base continuously accumulates the semantic patterns and expression habits of the current conference, and the accuracy of ambiguity correction gradually improves. After the conference ends, the ternary association network constructed in this conference and its association with standard vocabulary are stored in the historical conference database, serving as the initial knowledge base for subsequent conferences with the same topic or the same participants, enabling cross-conference transfer and reuse of contextual knowledge.

[0116] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the audio segmentation and sound source identification association process according to an embodiment of the present invention.

[0117] Extract the energy envelope of each audio segment, determine the segmentation point based on the valley position of the energy envelope, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a set of subtitle units including:

[0118] Extract the energy envelope of each audio sub-segment to detect the valley position, and extract the corresponding text of each valley position in the correction label text set to construct a dependency relationship chain;

[0119] The ratio of the number to the length of dependency chains cut off at the trough position is used as the syntactic split degree. Trough positions with a syntactic split degree lower than a preset split threshold are selected, and the split point is determined based on the energy envelope trough position.

[0120] Historical speech texts for each sound source identifier are extracted from historical conference data to construct a sound source feature word list and word sequence pattern. The feature word coverage and word sequence matching degree of the window texts before and after each segmentation point in the corrected identifier text set are calculated under each sound source identifier.

[0121] The difference between coverage and matching degree of the same sound source identifier between adjacent segmentation points is calculated as the cross-segmentation point change amplitude, and the sound source identifier with the largest cross-segmentation point change amplitude is selected and associated with the segmentation point;

[0122] Based on the segmentation points, the corresponding text in the set of correction identifiers is segmented to obtain text fragments, and the coverage and matching degree of the text fragments under the associated sound source identifiers are calculated.

[0123] Text segments with coverage and matching accuracy below average are reassociated with the sound source identifiers with the highest coverage and matching accuracy. The sound source identifiers are kept associated, and the text segments are combined with the sound source identifiers and timestamps to generate a set of subtitle units.

[0124] In the process of converting audio segments into subtitle units, it is necessary to determine reasonable text segmentation positions and maintain the sound source attribution relationship. An energy envelope curve is extracted from the input audio segment; this envelope reflects the intensity change trend of the speech signal over time. After smoothing the energy envelope, its trough positions are detected; these troughs typically correspond to speech pauses or intonation transitions. The detected trough positions are mapped to a corrected set of labeled text to obtain the corresponding character index of the trough position in the text sequence.

[0125] For each trough position corresponding to a text index point, construct dependency chains within a 15-character range before and after that point. These dependency chains reflect syntactic dependencies between words, including subject-verb, verb-object, and attributive-head relationships. Count how many dependency chains are severed if segmentation is performed at that trough position, and record the number of severed chains. Simultaneously calculate the average length of these severed relationship chains. Length is defined as the number of words spanned by the relation chain. and ratio Syntactic fragmentation at this position Syntactic fragmentation A higher position means that splitting at this point would severely disrupt the sentence structure, while Lower positions are natural sentence breaks. Setting the break threshold to 0.4 filters out... The trough locations below this threshold are used as the set of candidate segmentation points.

[0126] Historical speech texts corresponding to each voice source identifier were extracted from a historical conference database. Commonly used words and their frequencies for each voice source were statistically analyzed to construct a voice source feature vocabulary. This vocabulary not only includes high-frequency words but also records the TF-IDF weight of each word, highlighting the unique vocabulary habits that distinguish this voice source from others. Simultaneously, the continuous occurrence patterns of words in the historical texts were analyzed, extracting bigram and triplet word sequences, and counting the occurrence frequency of each type of word sequence to form a voice source-specific word sequence pattern library. These word sequence patterns can characterize the expression habits and language style of this voice source.

[0127] For each candidate segmentation point in the calibration label text set, 10 words before and after the segmentation point are extracted as a context window. The words in the front window text are matched with the feature word lists of each sound source, and the ratio of the number of matched feature words to the total number of words in the window is calculated to obtain the feature word coverage rate of the front window for each sound source. ,in This represents the sound source identifier number. The coverage of the window is also calculated similarly. The word sequences in the front and back windows are extracted and matched with the word sequence pattern library for each sound source. The proportion of successfully matched sequences to the total number of extracted sequences is calculated to obtain the word sequence matching degree of the front window. Matching degree with the back window .

[0128] For the text interval between two adjacent segmentation points, calculate the coverage difference between the front and rear windows under each sound source identifier. and matching degree difference The coverage difference and matching degree difference are weighted and summed, with weighting coefficients set to 0.6 and 0.4 respectively, to obtain the cross-segment variation range of the sound source at this segmentation point. A large variation indicates a significant difference in the linguistic features of the text before and after the segmentation point, possibly indicating a speaker switching position. (Selecting all sound sources...) The largest sound source identifier is associated with the current segmentation point, serving as the source of the text segment following that segmentation point.

[0129] Based on all segmentation point locations, the set of corrected identifier text is divided into multiple text segments. Each text segment corresponds to the text content between two adjacent segmentation points, or the content from the start of the text to the first segmentation point. For each text segment, the associated sound source identifiers have been obtained from the preceding steps. The coverage rate of feature words in this text segment within the vocabulary corresponding to the associated sound source identifiers is calculated. Simultaneously, word sequences are extracted from the text fragments and matched with a word sequence pattern library associated with the sound sources to obtain the matching degree. .

[0130] Calculate the coverage and matching degree of all text segments, and then calculate the average coverage. and the average matching degree Traverse all text fragments and identify coverage. Below And matching degree Below The text fragments. For these fragments, the currently associated sound source identifiers may be inaccurate. Recalculate the coverage and matching degree of the text fragments under all sound source identifiers, and then sum the coverage and matching degree by weight. The highest sound source identifier As a new associated sound source for this text fragment, the sound source identifier is reassigned.

[0131] Each text segment is bound to its associated sound source identifier, and the timestamp information corresponding to that text segment, including start and end times, is extracted from the audio sub-segment. The text content, sound source identifier, and timestamp are combined to form a subtitle unit. After all text segments have undergone the above processing, a set of subtitle units containing multiple subtitle units is formed, with each subtitle unit clearly identifying the speaker's identity, speech content, and time information.

[0132] In practical applications, if a meeting involves three speakers, historical data shows that the first speaker frequently uses words like "therefore," "in summary," and "data shows," the second speaker prefers expressions like "considering," "based on," and "recommended," while the third speaker frequently uses phrases like "specifically," "for example," and "in practice." When the text before the segmentation point reads "data shows sales growth" and the text after the segmentation point reads "considering market changes," the feature word coverage rate of the front window reaches 0.75 for the first speaker's voice source, and the coverage rate of the back window reaches 0.82 for the second speaker's voice source. Calculations show that the second speaker's voice source exhibits the largest cross-segmentation point variation, and this segmentation point is associated with the second speaker's voice source, indicating a speaker switch at this point.

[0133] For a segmented text that "requires further analysis," if the segment is currently associated with the first speaker's voice source, but its coverage is only 0.35 and its matching degree is 0.28, both lower than the averages of 0.58 and 0.52, the metrics for this segment are recalculated across all voice sources. It is found that the coverage reaches 0.71 and the matching degree is 0.68 with the third speaker's voice source, resulting in the highest overall score. The voice source identifier for this text segment is then reassigned to the third speaker, correcting the initial assignment error. Through this process, each caption unit is ensured to be both semantically complete and accurately reflect the speaker's identity, providing reliable basic data for subsequent caption display.

[0134] The display time is calculated based on the duration of the audio sub-segment corresponding to each subtitle unit in the subtitle unit set. Subtitle units with overlapping display times are identified, and vertical offsets are allocated according to differences in sound source identifiers. A spatial separation scheme is generated, including:

[0135] Extract the duration and start timestamp of the audio sub-segment corresponding to each subtitle unit from the subtitle unit set to calculate the display time;

[0136] Extract role labels and speaking weights of each sound source identifier from historical conference data, calculate the speaking weight difference of each sound source identifier in the display time overlapping subtitle unit pair, and identify the main speaking subtitle unit and the secondary speaking subtitle unit in the display time overlapping subtitle unit pair based on the speaking weight difference.

[0137] Detect whether the start timestamp of the secondary speech caption unit is later than the start timestamp of the main speech caption unit, and mark the secondary speech caption unit whose start timestamp is later than the main speech caption unit as an interjection caption unit;

[0138] A spatial separation scheme is generated by assigning a center vertical offset to the main speech caption unit, an edge vertical offset to the interstitial speech caption unit, and a symmetrical vertical offset to the non-interstitial secondary speech caption unit.

[0139] When generating a spatial separation scheme for subtitles with different speakers, it is necessary to comprehensively consider the temporal attributes of the subtitle units, speaker characteristics, and the rationality of the visual presentation. By accurately calculating the duration and timestamp of the audio sub-segments, the display time range of each subtitle unit in the video frame can be determined. In specific implementation, the metadata of the audio sub-segment associated with each subtitle unit is read one by one from the subtitle unit set. This metadata includes the start timestamp of the audio sub-segment. and duration Start timestamp It records the starting position of the audio segment in the original audio stream, usually expressed in milliseconds, and its duration. This reflects the time span of the audio segment from beginning to end. Based on these two time parameters, the display time interval of the subtitle unit can be calculated. This range defines the full time frame during which a caption unit must remain visible in the video frame.

[0140] To achieve a more intelligent captioning strategy, it's necessary to mine role information and speaking characteristics corresponding to each voice source identifier from historical meeting data. The historical meeting database stores speaking records of each participant in past meetings, including multi-dimensional statistical information such as speaking frequency, total speaking duration, number of interruptions, and frequency of responses. By analyzing this historical data, role tags can be generated for each voice source identifier, such as meeting host, key speaker, ordinary participant, and observer. Simultaneously, a speaking weight value is assigned to each voice source identifier based on a weighted calculation of speaking frequency and speaking duration. This weight value reflects the speaker's influence and importance in the meeting. The calculation of the speaking weight can take into account factors such as the proportion of total speaking time, the response rate of the speaking, and the continuity of the speaking, to form a normalized weight score, which usually ranges from 0 to 1.

[0141] When the display time intervals of multiple subtitle units overlap, a spatial separation strategy is needed to avoid visual conflicts in the subtitle content. The process of detecting overlapping display times employs an interval intersection judgment method. For any two subtitle units, if their display time intervals intersect—that is, the end time of the first subtitle unit is greater than the start time of the second subtitle unit, and the end time of the second subtitle unit is greater than the start time of the first subtitle unit—then these two subtitle units are considered to overlap in time, forming a subtitle unit pair. For each identified subtitle unit pair, its associated sound source identifier is extracted, and the corresponding speech weight values ​​for these two sound source identifiers are retrieved from historical data. and Calculate the difference in speaking weights Based on the weight of each speaker's remarks, subtitle units with higher weights are marked as primary speaker subtitle units, while those with lower weights are marked as secondary speaker subtitle units. This weighted approach visually highlights the content of important speakers, aligning with the information hierarchy in a meeting setting.

[0142] After determining the primary and secondary speaking relationships, the temporal characteristics of the secondary speaking caption units are further analyzed to identify interruption behavior. This is done by comparing the start timestamps of the secondary speaking caption units. Start timestamp of the main speaker caption unit If the start time of the next speech caption unit is later than that of the main speech caption unit, that is... This indicates that the second speaker intervenes after the main speaker has begun speaking. In a meeting setting, this typically corresponds to interrupting, providing supplementary information, or asking impromptu questions. Subtitle units that meet this timing condition are marked as interruption subtitle units, and this marking will influence subsequent vertical position allocation strategies. Identifying interruption subtitle units is crucial for creating a realistic meeting atmosphere, as it allows for the reconstruction of the interaction and dialogue dynamics between speakers in the subtitle presentation.

[0143] Based on the hierarchy of subtitle units and the characteristics of interludes, a differentiated vertical offset allocation scheme is formulated. For the main subtitle unit, a center vertical offset is allocated, positioning it in the lower-center area of ​​the video frame, typically set at 70% to 80% of the frame height. This position ensures readability without excessively obscuring the core area of ​​the video content, conforming to the viewer's visual habits. For identified interlude subtitle units, an edge vertical offset is allocated, positioning them in the upper or side areas of the frame, for example, setting the vertical coordinate at 20% to 30% of the frame height. By placing the interlude content in the visual edge area, its visibility is maintained while minimizing interference with the main content, forming a clear visual hierarchy. For non-interlude secondary subtitle units, i.e., those whose start time is earlier than or equal to the start time of the main subtitle unit, a symmetrical vertical offset allocation strategy is adopted. These types of subtitle units typically correspond to scenarios of parallel or alternating speech. They are assigned a vertical position that is symmetrical to the main speech subtitle unit. For example, when the main speech subtitle is located at the bottom center of the screen, the secondary speech subtitle can be located at the top center of the screen, maintaining the same vertical distance from the main speech subtitle but in the opposite direction.

[0144] The specific calculation of the vertical offset needs to take into account the subtitle height and the safe area of ​​the screen. Assume the standard height of a single-line subtitle is... Then the vertical coordinate of the main speaker caption unit can be set to ,in This is the total height of the video frame. The vertical coordinates of the interlude subtitle unit are set to... Ensure it is positioned above the main text and maintains sufficient vertical spacing. The vertical coordinates of the symmetrical secondary text unit are calculated as follows: This achieves a mirror-symmetric layout about the center line of the screen. When three or more subtitle units overlap at the same time, the vertical offset strategy is further refined, and the vertical positions are allocated from the center to the edge in descending order of speech weight, ensuring that all subtitle units have independent display space in the vertical direction.

[0145] The generated spatial separation scheme stores the spatial attributes of each subtitle unit in the form of a data structure, including parameters such as subtitle unit identifier, associated sound source identifier, display time interval, vertical offset, and horizontal alignment. This scheme serves as a positional reference for subsequent subtitle rendering, ensuring that subtitles in multi-person speaking scenarios can be clearly distinguished and presented in an orderly manner, avoiding information confusion. Through this spatial separation strategy based on speaking weight and temporal relationships, it is possible to convey the role relationships and dialogue dynamics in the meeting while maintaining subtitle readability, thereby improving the intelligence level of live conference subtitles and the quality of user experience.

[0146] Extracting the foreground target area of ​​the video frame to determine the subtitle-disallowed area, adjusting the display coordinates of the subtitle unit according to the subtitle-disallowed area and spatial separation scheme, and rendering it to the video frame to complete the subtitle output includes:

[0147] Semantic segmentation is performed on video frames to extract foreground target regions. The inter-frame displacement vectors and gradient consistency of pixels within the foreground target regions are calculated to construct a motion texture fusion matrix. Morphological dilation is performed on the motion texture fusion matrix to determine the subtitle-prohibited regions.

[0148] The vertical offset and sound source identifier of the subtitle unit are extracted from the spatial separation scheme. The display coordinates of the sound source identifier in the historical video frames are extracted to construct the sound source position probability distribution map. The initial display coordinates are calculated based on the probability peak and vertical offset of the sound source position probability distribution map.

[0149] Calculate the percentage of the intersection area between the initial display coordinates and the subtitle disabled area, filter subtitle units whose intersection area percentage exceeds the threshold as conflicting subtitle units, and sort them according to the weight of the number of words spoken;

[0150] Search for candidate display coordinates outside the subtitle-disabled area. Calculate the ratio of the probability density of the candidate display coordinates in the probability distribution map of the sound source location to the distance to the boundary of the subtitle-disabled area as a position stability index. Select the candidate display coordinates with the largest position stability index to replace the initial display coordinates to obtain the adjusted display coordinates.

[0151] The subtitle unit is rendered to the video frame based on the adjusted display coordinates to complete the subtitle output. The adjusted display coordinates are then accumulated to the probability distribution map of the corresponding sound source location.

[0152] After generating the spatial separation scheme for the subtitle unit, the subtitles need to be rendered onto the video screen. Considering that live conference footage typically contains important visual information such as the speaker's face, shared screen content, and presentation slides, subtitles obscuring this content will severely impact the viewing experience. To achieve intelligent obstacle avoidance and stable subtitle display, semantic segmentation is first performed on the current video frame, using a deep learning segmentation model to identify foreground targets such as face regions, document regions, and chart regions. The semantic segmentation process outputs pixel-level category labels, extracting pixel regions belonging to important foreground targets to form candidate banned regions. Relying solely on the semantic segmentation results of a single frame carries the risk of misjudgment; textures in a static background may be misidentified as foreground, while features of actual moving targets are more reliable.

[0153] For the extracted foreground target region, the displacement vector of each pixel within that region is calculated across consecutive video frames. Using optical flow algorithms or block matching methods, the motion vector of each pixel from the previous frame to the current frame is obtained; this vector reflects the target's motion trend. Simultaneously, the gradient consistency of each pixel is calculated, i.e., the stability of its gradient direction over time. The edge gradients of moving targets maintain a relatively consistent direction across consecutive frames, while the gradients of noise or misjudged regions exhibit random fluctuations. The magnitudes of the inter-frame displacement vectors and the gradient consistency metric are weighted and fused to construct a motion texture fusion matrix. Each element of this matrix corresponds to a pixel position in a video frame; a larger value indicates a higher confidence that the position belongs to a real moving foreground target. The motion texture fusion matrix is ​​binarized by setting a threshold, retaining high-confidence regions as initial unchecked caption areas.

[0154] Considering that subtitle text has a certain width and height, if the subtitle boundary is close to the edge of the foreground target, even slight target movement or segmentation errors could lead to occlusion. A morphological dilation operation is performed on the initially determined subtitle-free area, using rectangular or elliptical structuring elements to expand outwards by several pixels from the boundary of the free area. The kernel size of the dilation operation is dynamically adjusted according to the subtitle font size; the larger the font size, the larger the dilation range, ensuring sufficient safety distance between the subtitle and the foreground target. The area after dilation is the final subtitle-free area, which will be strictly avoided in subsequent subtitle layouts.

[0155] The vertical offset and associated sound source identifier of the current subtitle unit to be rendered are extracted from the spatial separation scheme generated in the previous steps. The vertical offset reflects the preset vertical position adjustment value of the subtitle unit to avoid overlapping with other sound source subtitles at the same time. The sound source identifier is used to track the display position history of a specific speaker's subtitles. The display coordinate records of the subtitles corresponding to the sound source identifier in historical video frames are retrieved. These historical coordinates reflect the permanent display area of ​​the speaker's subtitles. The historical display coordinates are statistically distributed in the video frame plane to construct a two-dimensional probability distribution map of the sound source position. This distribution map uses the kernel density estimation method, assigning Gaussian weights around each historical coordinate point, and superimposing them to form a smooth probability density surface. Areas with high probability density indicate the positions where the sound source subtitle has been frequently displayed in history, and viewers have formed visual habits.

[0156] The probability peak position is searched in the sound source location probability distribution map; this position represents the most stable display area for the subtitles of that sound source. The initial display coordinates are calculated by combining the horizontal coordinates of the probability peak position with the vertical offset obtained from the spatial separation scheme. These initial display coordinates take into account historical display habits and current sound source separation requirements, and ideally should represent the final position of the subtitles. In real-world scenarios, the initial display coordinates may fall into subtitle-disabled areas, requiring further detection and adjustment.

[0157] The intersection of the initial display coordinates-defined subtitle rectangle and the subtitle disabled area is calculated. The subtitle rectangle is calculated based on the subtitle text length and font parameters, and the intersection area reflects the degree to which the subtitle occludes the foreground target. The proportion of the intersection area to the total area of ​​the subtitle rectangle is calculated. When this proportion exceeds a preset threshold, the subtitle unit is considered to have a significant occlusion risk and is marked as a conflicting subtitle unit. For multiple conflicting subtitle units, they are sorted according to the speaking volume weight of their respective sound sources. The speaking volume weight is calculated by statistically analyzing the cumulative speaking time of the sound source in the current meeting. Sound sources with higher speaking volume have higher subtitle adjustment priority, and suitable display positions are found for them first.

[0158] For conflicting subtitle units, candidate display coordinates are searched in the space surrounding the subtitle prohibition area. The search strategy employs a spiral path or grid sampling method that expands outward from the initial display coordinates to ensure that candidate positions cover the available space around the prohibition area. Each candidate display coordinate undergoes a dual evaluation: first, the probability density value corresponding to the coordinate in the sound source location probability distribution map is queried; a higher probability density indicates that the position is closer to the viewer's visual expectation. Second, the minimum distance from the boundary of the subtitle rectangle formed by the candidate coordinates to the boundary of the nearest subtitle prohibition area is calculated; a larger distance indicates a lower risk of the subtitle being affected by the movement of foreground targets, and a more stable display position. The ratio of the probability density value to the distance value is used as a position stability index, which prioritizes display areas familiar to the viewer while ensuring that the subtitle is not obscured.

[0159] All candidate display coordinates are traversed, and the coordinates with the highest positional stability index are selected as the optimal candidate. The original initial display coordinates are replaced with these optimal candidate coordinates to obtain the adjusted display coordinates. The adjusted display coordinates, while meeting the requirement of avoiding prohibited areas, maintain the continuity of the display position of the subtitles for that sound source to the greatest extent possible, reducing the abruptness of the subtitles on the screen. For conflicting subtitle units with low speech weight, after the position of the main speaker's subtitle is determined, display coordinates are allocated sequentially in the remaining available space to ensure that all subtitle units receive a reasonable display position.

[0160] Based on the adjusted display coordinates, the text content of the subtitle unit is rendered to the corresponding position in the current video frame according to the preset font, color, and border style. Anti-aliasing technology is used during rendering to ensure smooth text edges, and stroke or shadow effects are added to enhance the readability of the subtitles against complex backgrounds. The rendered video frame contains intelligently laid-out subtitle information, avoiding obscuring important visual content while maintaining the relative stability of the subtitle positions for each sound source. The adjusted display coordinates actually used in the current frame are recorded in the historical coordinate database of the corresponding sound source identifier, and the position probability distribution map of that sound source is updated. The accumulated historical data allows the sound source position probability distribution map to dynamically evolve as the meeting progresses, reflecting the true distribution characteristics of the subtitle display positions for each speaker, providing a more accurate reference for the subtitle layout of subsequent video frames. Through continuous position learning and optimization, the subtitle display gradually converges to a stable spatial layout pattern, improving viewer comfort and information acquisition efficiency.

[0161] A second aspect of this invention provides a real-time caption generation system for live video conferencing that integrates AI, the system comprising:

[0162] The audio processing unit is used to collect audio stream data and divide audio segments according to time windows, separate and identify each sound source in the audio segments and extract audio sub-segments, assign sound source identifiers to each audio sub-segment, and generate an identifier audio set.

[0163] The speech recognition unit is used to perform speech recognition on each audio sub-segment in the identified audio set and associate it with the sound source identifier to generate an identified text set.

[0164] The text correction unit is used to construct a contextual knowledge base containing a thesaurus of conference topics and historical texts. Based on the contextual knowledge base, it matches and corrects ambiguous words in the identified text set to generate a corrected identified text set.

[0165] The text segmentation unit is used to extract the energy envelope of each audio sub-segment, determine the segmentation point based on the position of the energy envelope trough, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a set of subtitle units.

[0166] The spatial allocation unit is used to calculate the display time based on the duration of the audio sub-segments corresponding to each subtitle unit in the subtitle unit set, identify subtitle units with overlapping display times, allocate vertical offsets based on differences in sound source identifiers, and generate a spatial separation scheme.

[0167] The subtitle rendering unit is used to extract the foreground target area of ​​the video frame to determine the subtitle disabled area, adjust the display coordinates of the subtitle unit according to the subtitle disabled area and spatial separation scheme, and render it to the video frame to complete the subtitle output.

[0168] A third aspect of the present invention provides an electronic device, comprising:

[0169] processor;

[0170] Memory used to store processor-executable instructions;

[0171] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0172] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0173] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating real-time subtitles for live video conferencing by integrating AI, characterized in that, include: Collect audio stream data and divide audio segments according to time windows. Separate and identify the sound sources of each audio segment and extract audio sub-segments. Assign sound source identifiers to each audio sub-segment and generate an identifiable audio set. Speech recognition is performed on each audio sub-segment in the identified audio set, and sound source identifiers are associated with them to generate an identified text set; Construct a contextual knowledge base containing a thesaurus of conference topics and historical texts. Based on the contextual knowledge base, match and correct ambiguous words in the identified text set to generate a corrected identified text set. Extract the energy envelope of each audio sub-segment, determine the segmentation point based on the position of the energy envelope trough, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a subtitle unit set; The display time is calculated based on the duration of the audio sub-segment corresponding to each subtitle unit in the subtitle unit set. Subtitle units with overlapping display times are identified and vertical offsets are allocated according to the differences in sound source identifiers to generate a spatial separation scheme. Extract the foreground target area of ​​the video frame to determine the subtitle ban area, adjust the display coordinates of the subtitle unit according to the subtitle ban area and spatial separation scheme, and render it to the video frame to complete the subtitle output.

2. The method according to claim 1, characterized in that, The audio stream data is acquired and divided into audio segments according to time windows. Sound sources are separated and identified within each audio segment, and audio sub-segments are extracted. Sound source identifiers are assigned to each audio sub-segment, generating an identifiable audio set including: The system collects audio stream data and detects energy changes in the audio stream data. Based on the energy changes, it identifies silent intervals and active intervals. The silent intervals are used as the boundaries of audio segments, and the active intervals are divided into audio segments according to time windows. The audio segment is subjected to short-time Fourier transform to obtain the time-frequency matrix. The harmonic structure and formant distribution are extracted from the time-frequency matrix to construct the sound source mask matrix. The time-frequency matrix and the sound source mask matrix are multiplied element by element to separate the time-frequency components of each sound source. The audio sub-segments are reconstructed by performing inverse short-time Fourier transform on the time-frequency components of each sound source. The Mel-frequency cepstral coefficients of each audio sub-segment are extracted and combined with the fundamental frequency trajectory to form a voiceprint descriptor. The Mahalanobis distance between the voiceprint descriptors is calculated and spectral clustering is performed to obtain the sound source clustering results. Assign a unique sound source identifier to the cluster center in the sound source clustering results, calculate the signal-to-noise ratio distribution of the audio sub-segment corresponding to each sound source identifier, assign a confidence weight to each sound source identifier according to the signal-to-noise ratio distribution, associate the sound source identifier with the confidence weight to the corresponding audio sub-segment, and generate an identifier audio set.

3. The method according to claim 1, characterized in that, Speech recognition is performed on each audio sub-segment in the tagged audio set, and sound source identifiers are associated with them to generate a tagged text set, including: Extract each audio sub-segment and its corresponding sound source identifier from the identified audio set. Perform phoneme segmentation on each audio sub-segment to obtain a phoneme sequence. Perform forward and backward traversal on the phoneme sequence to obtain forward and backward dependencies. Merge the forward and backward dependencies to obtain a bidirectional dependency. Based on the bidirectional dependency relationship, the phoneme sequence is combined into multiple candidate texts, and the degree of matching between each candidate text and the phoneme sequence is calculated to obtain the acoustic score; The language score is obtained by calculating the transition probabilities between words within each candidate text. The acoustic score and the language score are weighted and summed to obtain a comprehensive score. The candidate text with the highest comprehensive score is selected as the recognition text. The sound source identifiers of each recognized text and the corresponding audio sub-segment are associated. The number of characters in the recognized text corresponding to each sound source identifier and the duration of the audio sub-segment are counted. The speech rate value of each sound source identifier is calculated based on the number of characters and the duration. Based on the speech rate value, a speech volume weight is assigned to each sound source identifier, and the identified text, sound source identifier, and speech volume weight are combined to generate an identifier text set.

4. The method according to claim 1, characterized in that, A contextual knowledge base containing a thesaurus of conference topics and historical texts is constructed. Based on the contextual knowledge base, ambiguous words in the identified text set are matched and corrected, generating a corrected identified text set including: Extract core keywords from the identified text set to construct a conference keyword list; extract speech texts from historical conference data as historical texts to construct a sound source semantic profile; identify topic switching points from the identified text set to extract core words from topic segments to construct topic segment context features. A ternary association network is established by combining the conference theme thesaurus, the semantic profile of the sound source, and the contextual features of the theme segment. The ternary association network is then combined with historical texts to construct a contextual knowledge base that includes the conference theme thesaurus and historical texts. The text is extracted from the identified text set to detect ambiguous words and extract local context. Based on the context knowledge base, the shortest path length of the local context and each semantic interpretation of the ambiguous word in the ternary association network is calculated to obtain the context tightness. The number of paths through each semantic interpretation is counted to obtain the semantic support. The context tightness and semantic support are combined to calculate the context dependency strength. Based on the context knowledge base, ambiguous words in the identified text set are matched and corrected. The semantic interpretation with the strongest context dependence is selected to replace the ambiguous word to obtain the corrected text. The corrected text is combined with the corresponding sound source identifier to generate a set of corrected identifier texts. The standard words in the corrected text are backpropagated to update the ternary association network to obtain an updated contextual knowledge base, which is used for the correction of ambiguous words in the identifier text set in the next time window.

5. The method according to claim 1, characterized in that, Extract the energy envelope of each audio segment, determine the segmentation point based on the valley position of the energy envelope, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a set of subtitle units including: Extract the energy envelope of each audio sub-segment to detect the valley position, and extract the corresponding text of each valley position in the correction label text set to construct a dependency relationship chain; The ratio of the number to the length of dependency chains cut off at the trough position is used as the syntactic split degree. Trough positions with a syntactic split degree lower than a preset split threshold are selected, and the split point is determined based on the energy envelope trough position. Historical speech texts for each sound source identifier are extracted from historical conference data to construct a sound source feature word list and word sequence pattern. The feature word coverage and word sequence matching degree of the window texts before and after each segmentation point in the corrected identifier text set are calculated under each sound source identifier. The difference between coverage and matching degree of the same sound source identifier between adjacent segmentation points is calculated as the cross-segmentation point change amplitude, and the sound source identifier with the largest cross-segmentation point change amplitude is selected and associated with the segmentation point; Based on the segmentation points, the corresponding text in the set of correction identifiers is segmented to obtain text fragments, and the coverage and matching degree of the text fragments under the associated sound source identifiers are calculated. Text segments with coverage and matching accuracy below average are reassociated with the sound source identifiers with the highest coverage and matching accuracy. The sound source identifiers are kept associated, and the text segments are combined with the sound source identifiers and timestamps to generate a set of subtitle units.

6. The method according to claim 1, characterized in that, The display time is calculated based on the duration of the audio sub-segment corresponding to each subtitle unit in the subtitle unit set. Subtitle units with overlapping display times are identified, and vertical offsets are allocated according to differences in sound source identifiers. A spatial separation scheme is generated, including: Extract the duration and start timestamp of the audio sub-segment corresponding to each subtitle unit from the subtitle unit set to calculate the display time; Extract role labels and speaking weights of each sound source identifier from historical conference data, calculate the speaking weight difference of each sound source identifier in the display time overlapping subtitle unit pair, and identify the main speaking subtitle unit and the secondary speaking subtitle unit in the display time overlapping subtitle unit pair based on the speaking weight difference. Detect whether the start timestamp of the secondary speech caption unit is later than the start timestamp of the main speech caption unit, and mark the secondary speech caption unit whose start timestamp is later than the main speech caption unit as an interjection caption unit; Assign a center vertical offset to the main speech caption unit, assign an edge vertical offset to the interstitial speech caption unit, and assign a symmetrical vertical offset to the non-interstitial secondary speech caption unit to generate a spatial separation scheme.

7. The method according to claim 1, characterized in that, Extracting the foreground target area of ​​the video frame to determine the subtitle-disallowed area, adjusting the display coordinates of the subtitle unit according to the subtitle-disallowed area and spatial separation scheme, and rendering it to the video frame to complete the subtitle output includes: Semantic segmentation is performed on video frames to extract foreground target regions. The inter-frame displacement vectors and gradient consistency of pixels within the foreground target regions are calculated to construct a motion texture fusion matrix. Morphological dilation is performed on the motion texture fusion matrix to determine the subtitle-prohibited regions. The vertical offset and sound source identifier of the subtitle unit are extracted from the spatial separation scheme. The display coordinates of the sound source identifier in the historical video frames are extracted to construct the sound source position probability distribution map. The initial display coordinates are calculated based on the probability peak and vertical offset of the sound source position probability distribution map. Calculate the percentage of the intersection area between the initial display coordinates and the subtitle disabled area, filter subtitle units whose intersection area percentage exceeds the threshold as conflicting subtitle units, and sort them according to the weight of the number of words spoken; Search for candidate display coordinates outside the subtitle-disabled area. Calculate the ratio of the probability density of the candidate display coordinates in the probability distribution map of the sound source location to the distance to the boundary of the subtitle-disabled area as a position stability index. Select the candidate display coordinates with the largest position stability index to replace the initial display coordinates to obtain the adjusted display coordinates. The subtitle unit is rendered to the video frame based on the adjusted display coordinates to complete the subtitle output. The adjusted display coordinates are then accumulated to the probability distribution map of the corresponding sound source location.

8. A live-streaming conference real-time caption generation system integrating AI, used to implement the method of any one of claims 1-7, characterized in that, include: The audio processing unit is used to collect audio stream data and divide audio segments according to time windows, separate and identify each sound source in the audio segments and extract audio sub-segments, assign sound source identifiers to each audio sub-segment, and generate an identifier audio set. The speech recognition unit is used to perform speech recognition on each audio sub-segment in the identified audio set and associate it with the sound source identifier to generate an identified text set. The text correction unit is used to construct a contextual knowledge base containing a thesaurus of conference topics and historical texts. Based on the contextual knowledge base, it matches and corrects ambiguous words in the identified text set to generate a corrected identified text set. The text segmentation unit is used to extract the energy envelope of each audio sub-segment, determine the segmentation point based on the position of the energy envelope trough, segment the corresponding text in the correction label text set according to the segmentation point while maintaining the sound source label association, and generate a set of subtitle units. The spatial allocation unit is used to calculate the display time based on the duration of the audio sub-segments corresponding to each subtitle unit in the subtitle unit set, identify subtitle units with overlapping display times, allocate vertical offsets based on differences in sound source identifiers, and generate a spatial separation scheme. The subtitle rendering unit is used to extract the foreground target area of ​​the video frame to determine the subtitle disabled area, adjust the display coordinates of the subtitle unit according to the subtitle disabled area and spatial separation scheme, and render it to the video frame to complete the subtitle output.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images