A video call caption real-time refreshing method based on voice large model analysis

Abstract

The application discloses a video call subtitle real-time refreshing method based on voice large model analysis, relates to the technical field of video calls, and comprises the following steps: extracting clear tone through time domain, frequency domain and sound spectrum graph analysis, performing natural language recognition and classification by using a voice large model, and embedding subtitle information and restoring a video stream by combining a timestamp matching algorithm.The application can effectively reduce noise interference, support multilingual and dialect subtitle generation, maintain the effect of real-time refreshing of subtitles under network delay, and significantly improve the communication experience of video calls.

Description

Technical Field

[0001] This invention belongs to the field of video call technology, specifically, it relates to a method for real-time refresh of video call subtitles based on large voice model analysis. Background Technology

[0002] With the rapid development of artificial intelligence technology, speech recognition technology has made significant progress, especially in foreign language translation and dialect processing, achieving high-quality conversion with accurate, expressive, and elegant speech. However, traditional video call technology still mainly focuses on the smoothness of network communication, failing to adequately address the real-time generation and display of subtitles during the call. Currently, subtitle applications are mainly concentrated in post-production scenarios such as movies and short videos, relying on human translation and editing, which cannot meet the needs of real-time subtitle assistance. In video calls, due to environmental noise, non-verbal sounds, and interference from abnormal speech, users often face problems with not being able to hear or understand, which is particularly prominent for certain groups. Furthermore, in cases of network lag, video frames and audio streams may not be transmitted normally, while subtitles, as text streams, have lower network dependence and can, to some extent, compensate for this deficiency, providing a more stable call experience.

[0003] Existing technologies have certain limitations in speech signal processing. For example, traditional speech recognition systems often cannot effectively distinguish between synthetic speech, environmental noise, non-verbal sounds, and abnormal speech, leading to decreased accuracy. Although methods such as spectral feature analysis, acoustic model adaptation, multi-task joint judgment, and contextual consistency checks can improve the robustness of speech recognition to some extent, these techniques are mostly designed for pre-recorded audio or specific scenarios and are difficult to adapt to the complex environment of real-time video calls. In addition, existing technologies lack the ability to accurately distinguish timbre and clarity, and cannot extract the most suitable speech data for subtitles from mixed audio signals, which further limits the possibility of its real-time application.

[0004] On the other hand, video call environments are filled with various interfering factors, such as machine noise, wind noise, coughing, and other non-verbal sounds, as well as abnormal speech patterns like whispers and screams, all of which can affect the accuracy of speech recognition. Furthermore, the audio signals in video calls often contain multiple languages, dialects, and even slang, placing higher demands on the language classification and conversion capabilities of large-scale speech models. Existing technologies typically lack sufficient flexibility and adaptability when handling these complex scenarios, making it difficult to achieve efficient and accurate real-time subtitle generation. Summary of the Invention

[0005] The purpose of this invention is to provide a method for real-time updating of video call subtitles based on large voice model analysis, which mainly solves the problems of lack of real-time subtitle function and susceptibility to noise interference in existing video call technologies.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for real-time video call caption refresh based on large speech model analysis includes the following steps:

[0008] S1, sample the original video call audio, and perform spectral processing on the sampled audio data to obtain a timbre that meets the set standard;

[0009] S2, input the extracted timbre into the large speech model to generate subtitle data;

[0010] S3, based on the subtitle timestamp matching algorithm and subtitle dwell time algorithm, embeds the generated subtitle data into the video stream of the original video call according to the video splitting and restoration protocol rules;

[0011] S4 refreshes the video stream with subtitles onto the player, enabling real-time updating of video call subtitles.

[0012] Furthermore, the specific process of step S1 is as follows:

[0013] S11, The sampled sound data is displayed through a comparison matrix to obtain the original speech signal waveform and the original speech signal spectrum;

[0014] S12, analyze the waveform of the original speech sound signal, obtain the change characteristics of the waveform in the time dimension, make a preliminary judgment on the clarity of the timbre, and display the analysis data according to the comparison matrix of data and feature values ​​to obtain the time domain analysis diagram composed of the time domain signal;

[0015] S13, converts the time-domain signal into a frequency-domain signal through Fourier transform, obtains a frequency-domain analysis graph, and identifies the timbre with frequencies concentrated within a set range;

[0016] S14, based on fundamental frequency perturbation, amplitude perturbation, and harmonic-to-noise ratio, obtains a timbre that meets the set standards in the spectrogram.

[0017] Furthermore, the specific steps of step S2 are as follows:

[0018] S21, the large speech model distinguishes between natural and non-natural language based on the input timbre;

[0019] S22: If the language is not natural, no subtitles will be output; if the language is natural, it will be classified as natural language.

[0020] S23 matches the classified natural language with the corresponding speech model, outputs the text of standard natural speech, and segments the text into sentences.

[0021] Furthermore, the specific process of step S3 is as follows:

[0022] S31, Read instantaneous images from the video stream of the original video call, and attach timestamp index information to each image to match the time information of the real-time captions;

[0023] S32, calculate the speed of the subtitle data, obtain the real-time subtitle information of each image, and add a timestamp index to each subtitle. Then, match the timestamp with each image in turn. After a successful match, print the subtitle information corresponding to the instantaneous image.

[0024] S33 inserts the processed instant image back into the original video stream, restoring it into a playable video.

[0025] Furthermore, in step S3, the implementation process of the subtitle timestamp matching algorithm is as follows:

[0026] Construct a caption data list, with the starting time set to x, and generate a caption list containing records for each image. Each record for each image contains a timestamp and corresponding text.

[0027] Starting from time x, the original video stream is stopped from being refreshed and continues for a specified duration. At the same time, a composite video stream is started at time x. Every time interval △t, a frame of instantaneous image is acquired from the original video stream. The process ends after acquiring a total of N frames. That is, the timestamp of the nth record is x + n×△t; n=0,1,...,N, covering the time range from x to x+1s; N is the total number of instantaneous images read, and △t is the instantaneous image reading interval.

[0028] Iterate through the caption list. If the difference between the timestamp of the current instant image and the timestamp of the nth record in the caption data array is less than a set value, then combine the current caption data into the current instant image.

[0029] Furthermore, in the subtitle timestamp matching algorithm, if the video stream refresh is abnormal due to network latency, the system enters a preset waiting time p seconds. If the normal video stream is restored within p seconds, the subtitle timestamp matching algorithm continues to be executed. If the normal video stream is not restored within p seconds, the waiting stops, and the currently acquired one frame of video stream is combined with the subtitle information and pushed for playback to ensure the subtitle refresh effect.

[0030] Furthermore, in step S3, the implementation process of the subtitle duration algorithm is as follows:

[0031] The duration A of the subtitle display is used as a system variable for external adjustment. If no new audio output is generated within a certain time period, the subtitle display will remain for a duration of A. If an audio output is generated during this period, the subtitle timestamp matching algorithm will be initiated immediately.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) This invention extracts clear timbre from the original speech through time domain analysis (waveform feature extraction), frequency domain analysis (Fourier transform) and spectrogram analysis (fundamental frequency perturbation and harmonic noise ratio screening), and uses a large speech model to filter non-natural language (such as noise and synthesized speech), effectively reducing the impact of environmental interference on the accuracy of subtitles.

[0034] (2) The present invention uses a subtitle timestamp matching algorithm to split the video stream into instantaneous images and accurately align them with the subtitle timestamps (e.g., matching 24 frames per second with the corresponding subtitles). Combined with a preset waiting mechanism (the current frame is synthesized after p seconds when there is network delay), the subtitles can still be refreshed in real time under network fluctuations, thus improving the smoothness of video calls.

[0035] (3) The speech model in this invention can distinguish between natural language and non-natural language, and match standard speech model or dialect / slang model (such as through professionally labeled dialect training data) according to the classification results to realize the generation of multilingual and dialect subtitles, meet the needs of cross-border and cross-regional communication, and expand the application scenarios. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the speech large model conversion process in this invention.

[0037] Figure 2 This is a schematic diagram of the subtitle display process in this invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0039] like Figure 1 As shown, the present invention discloses a method for real-time refresh of video call subtitles based on large voice model analysis, which realizes real-time generation and display of subtitles through steps such as timbre identification, large voice model conversion, timestamp matching and video stream restoration.

[0040] In practical applications, such as during cross-border video conferences, users A and B communicate in different languages, and both want to see real-time subtitles in the other's language for better understanding. In this scenario, the system first extracts the audio signal from the original video call and performs time-domain, frequency-domain, and spectrogram analysis to identify the clearest timbre. Through time-domain analysis, the system filters waveforms with a frequency range of 85Hz to 255Hz and amplitude variations stable between 30 and 70 dB as preliminary criteria. Subsequently, Fourier transform is used to convert the time-domain signal to a frequency-domain signal, observing spectral characteristics and locking in timbres concentrated between 85Hz and 255Hz. Further, spectrogram analysis, combined with fundamental frequency perturbation, amplitude perturbation, and harmonic-to-noise ratio (HNR) metrics, further filters the human voice waveform. The spectrogram is obtained by performing a short-time Fourier transform on the original audio signal to obtain the frequency component intensity for each short time interval. After several short-time Fourier transforms, the time intervals are finally stitched together to obtain the spectrogram. The spectrogram differs from time-domain and frequency-domain analysis graphs; it is an analytical method that integrates time, frequency, and signal intensity dimensions. In this embodiment, the typical value range for fundamental frequency perturbation is 1.0% to 2.5%, the typical value range for amplitude perturbation is 4% to 6%, and the typical value range for harmonic noise ratio is 8 to 10 dB. After completing the above analysis, the system locks the clearest timbre for subsequent processing.

[0041] Next, the system enters the speech model conversion stage. The extracted clear timbre is input into the OpenAI Whisper speech model for natural language recognition and classification. The speech model distinguishes between natural and non-natural language through spectral feature analysis, acoustic model adaptation, multi-task joint judgment, and contextual consistency checks. For non-natural language such as synthesized speech, environmental noise, non-verbal sounds, and abnormal speech, the system does not generate subtitles. For natural language, the system determines the speaker's language category based on the speech model's results and calls the corresponding standard natural language model or slang / dialect speech model for conversion. Slang / dialect training data is collected and annotated using professional language data acquisition tools to form fixed-format audio and text annotation files. The audio files are required to be 16kHz sampling rate, mono, and in WAV format, while the text annotations are required to be UTF-8 encoded and strictly aligned with the audio. After processing by the speech model, the system obtains real-time subtitle information.

[0042] like Figure 2As shown, after acquiring real-time subtitle information, the system begins to split the original video stream and print subtitles. During the splitting process, the system reads instantaneous images from the video stream at a rate of 24 per second, and assigns a timestamp index to each image for subsequent matching. Subsequently, the system generates 24 real-time subtitle entries based on the speed of the speech recognition results, and assigns a timestamp index to each subtitle. The specific implementation of the timestamp matching algorithm is as follows: The system waits for the speech recognition result to be output at the current moment, retrieves the speech recognition result within the next second, and synthesizes the result into a text segment as the subtitle information from a certain moment to the next second. Let the moment when subtitle information has been collected be x, and the collected subtitle information is stored as a list of length 24. Each subtitle record is used to match the timestamp of the instantaneous image. The timestamp of the first subtitle is x, the next is x+40ms, and so on until the last subtitle has a timestamp of x+1s. Starting from time x, the original video stream is stopped from being refreshed, and the synthesized video stream is refreshed starting from time x. Starting at time x, instantaneous images are acquired every 40ms from the original video stream, ending when 24 instantaneous images have been obtained. During image acquisition, the system matches timestamps with timestamps in the caption information list, using the following matching rules:

[0043] 1for (int zmIndex = 0; zmIndex <ZMData.size(); zmIndex++) {

[0044] 2 if (fabs(currentTime - ZMData[zmIndex].time)<0.0001) {

[0045] 3 / / Obtain a single frame of the current moment and synthesize it with the collected subtitle information.

[0046] 4 break;

[0047] 5}

[0048] Upon successful matching, the system immediately combines the current frame image with its corresponding subtitle information and pushes it to the player for playback. If network latency causes abnormal video stream refresh, the system processes the request after a preset waiting time of p seconds. If the video stream is successfully acquired within p seconds, the original algorithm continues; if no video stream is acquired, the system stops waiting for the latest video stream and instead combines the current frame video stream with the subtitle information, pushing it to the player for playback to achieve the subtitle refresh effect. The waiting time p can be used as a system variable for external adjustment.

[0049] The system also sets the subtitle display duration as a system variable for external adjustment. If no new audio output occurs within a time period, the subtitles remain for duration A; if new audio output occurs, the timestamp matching algorithm is immediately initiated. Video splitting and restoration protocol rules ensure real-time subtitle refresh and smooth playback. Following the original video refresh logic, the video stream from time x to x+1s will be refreshed onto the player normally. To achieve the subtitle effect, the system manually timestamps each entry in the subtitle list when collecting video streams within a time period, thus preparing subtitle information for the next second. The player stops refreshing the original video stream after the subtitle start time, waiting for the synthesized video stream with subtitle information to be interpolated back for playback. When the last frame is played, the system continues to connect the original video stream to the player. Within this one second, the subtitle information may have already been fully played in the previous frames; even in the worst case, where the subtitle information is only fully played in the last frame of 24 frames, it does not affect the subsequent connection of the original video stream to the player. This one second can be adjusted as a system variable to adapt to the impact of network latency.

[0050] Finally, the system reassembles the processed instantaneous images into a video stream, ensuring real-time subtitle refresh and smooth playback. During video frame synthesis, the system determines the subtitle dwell time and update time. If network latency causes abnormal video stream refresh, the system will synthesize the current frame of video stream and subtitle information after a preset waiting time of p seconds and push it to the player for playback. Ultimately, the system completes the video stream restoration operation and pushes the video stream with subtitle information to the player for playback, thus achieving the real-time subtitle refresh effect during video calls.

[0051] As can be seen from the above implementation process, this invention extracts clear timbre through time-domain, frequency-domain, and spectrogram analysis, significantly reducing the impact of noise interference on speech recognition. It utilizes large speech model technology to achieve natural language recognition and classification, generating accurate subtitle information. Through timestamp matching algorithms and subtitle dwell time control, it maintains real-time subtitle updates even with network latency. By integrating standard natural language models and slang / dialect speech models, it supports subtitle generation for multiple languages ​​and dialects, meeting communication needs in different scenarios.

[0052] The above embodiments are merely one of the preferred embodiments of the present invention and should not be used to limit the scope of protection of the present invention. Any modifications or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but solve the same technical problem as the present invention, should be included within the scope of protection of the present invention.

Claims

1. A method for real-time updating of video call subtitles based on large speech model analysis, characterized in that, Includes the following steps: S1, sample the original video call audio, and perform spectral processing on the sampled audio data to obtain a timbre that meets the set standards; the specific process is as follows: S11, The sampled sound data is displayed through a comparison matrix to obtain the original speech signal waveform and the original speech signal spectrum; S12, analyze the waveform of the original speech sound signal, obtain the change characteristics of the waveform in the time dimension, make a preliminary judgment on the clarity of the timbre, and display the analysis data according to the comparison matrix of data and feature values ​​to obtain the time domain analysis diagram composed of the time domain signal; S13, converts the time-domain signal into a frequency-domain signal through Fourier transform, obtains a frequency-domain analysis graph, and identifies the timbre with frequencies concentrated within a set range; S14, based on fundamental frequency perturbation, amplitude perturbation, and harmonic-to-noise ratio, obtains a timbre that meets the set standard in the spectrogram; S2, input the extracted timbre into the large speech model to generate subtitle data; S3, based on the subtitle timestamp matching algorithm and subtitle dwell time algorithm, embeds the generated subtitle data into the video stream of the original video call according to the video splitting and restoration protocol rules; S4 refreshes the video stream with subtitles onto the player, enabling real-time updating of video call subtitles.

2. The method for real-time video call subtitle refresh based on large speech model analysis according to claim 1, characterized in that, The specific steps of step S2 are as follows: S21, the large speech model distinguishes between natural and non-natural language based on the input timbre; S22: If the language is not natural, no subtitles will be output; if the language is natural, it will be classified as natural language. S23 matches the classified natural language with the corresponding speech model, outputs the text of standard natural speech, and segments the text into sentences.

3. The method for real-time video call subtitle refresh based on large speech model analysis according to claim 2, characterized in that, The specific process of step S3 is as follows: S31, Read instantaneous images from the video stream of the original video call, and attach timestamp index information to each image to match the time information of the real-time captions; S32, calculate the speed of the subtitle data, obtain the real-time subtitle information of each image, and add a timestamp index to each subtitle. Then, match the timestamp with each image in turn. After a successful match, print the subtitle information corresponding to the instantaneous image. S33 inserts the processed instant image back into the original video stream, restoring it into a playable video.

4. The method for real-time video call subtitle refresh based on large speech model analysis according to claim 3, characterized in that, The implementation process of the subtitle timestamp matching algorithm is as follows: Construct a caption data list, with the starting time set to x, and generate a caption list containing records for each image. Each record for each image contains a timestamp and corresponding text. Starting from time x, the original video stream is stopped from being refreshed and continues for a specified duration. At the same time, a composite video stream is started at time x. Every time interval △t, one instantaneous image is acquired from the original video stream. The process ends after acquiring a total of N images. That is, the timestamp of the nth record is x+n×△t; n=0,1,...,N, covering the time range from x to x+1s; N is the total number of instantaneous images read, and △t is the instantaneous image reading interval. Iterate through the caption list. If the difference between the timestamp of the current instant image and the timestamp of the nth record in the caption data array is less than a set value, then combine the current caption data into the current instant image.

5. The method for real-time video call subtitle refresh based on large speech model analysis according to claim 4, characterized in that, In step S3, in the subtitle timestamp matching algorithm, if the video stream refresh is abnormal due to network latency, the system enters a preset waiting time p seconds. If the normal video stream is restored within p seconds, the subtitle timestamp matching algorithm continues to be executed. If the normal video stream is not restored within p seconds, the waiting stops, and the currently acquired one frame of video stream is combined with the subtitle information and pushed for playback to ensure the subtitle refresh effect.

6. The method for real-time video call subtitle refresh based on large speech model analysis according to claim 5, characterized in that, In step S3, the implementation process of the subtitle duration algorithm is as follows: The duration A of the subtitle display is used as a system variable for external adjustment. If no new audio output is generated within a certain time period, the subtitle display will remain for a duration of A. If an audio output is generated during this period, the subtitle timestamp matching algorithm will be initiated immediately.

Images