Subtitle display method and device, equipment, storage medium and program product

Abstract

The invention discloses a subtitle display method and device, equipment, a storage medium and a program product, which are used for improving the subtitle display efficiency and accuracy and ensuring that subtitles are matched with the situation of a video picture. The method comprises the following steps: separating an audio stream and a video frame sequence from a video file; based on the audio features of the audio stream and the image features of each video frame in the video frame sequence, determining a target video frame synchronized with the voice signal in the audio stream in the video frame sequence; generating a subtitle text corresponding to the target video frame based on the voice signal corresponding to the target video frame; and displaying the subtitle text on the target video frame based on the time information of the voice signal.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, device, equipment, storage medium and program product for displaying subtitles during a call. Background Art

[0002] With the rapid development of the Internet and digital media technology, video content has become an important means for people to obtain information and entertainment, and the editing and transmission of video content has received more and more attention. Although the video content itself can provide visual and auditory information, the display of subtitles is indispensable in many cases. For example, when the sound cannot be played or the hearing-impaired watch the video, subtitles can provide necessary text information to help understand the video content. In addition, subtitles are also a key auxiliary means for non-native audiences to understand foreign language videos.

[0003] However, traditional subtitle display methods usually require a lot of manual intervention and mainly rely on manual dictation and editing. This process is time-consuming and error-prone, resulting in a mismatch between the subtitles and the context of the video. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a subtitle display method, device, equipment, storage medium and program product, which are used to improve the efficiency and accuracy of subtitle display and ensure that the subtitles match the context of the video screen.

[0005] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a subtitle display method, comprising:

[0007] Separate audio stream and video frame sequence from video files;

[0008] Determine, based on the audio features of the audio stream and the image features of each video frame in the video frame sequence, a target video frame in the video frame sequence that is synchronized with the voice signal in the audio stream;

[0009] Based on the speech signal corresponding to the target video frame, generating a subtitle text corresponding to the target video frame;

[0010] The subtitle text is displayed on the target video frame based on the time information of the speech signal.

[0011] In a second aspect, an embodiment of the present application provides a subtitle display device, comprising:

[0012] A separation module, used for separating audio stream and video frame sequence from a video file;

[0013] Determine, based on the audio features of the audio stream and the image features of each video frame in the video frame sequence, a target video frame in the video frame sequence that is synchronized with the voice signal in the audio stream;

[0014] Based on the speech signal corresponding to the target video frame, generating a subtitle text corresponding to the target video frame;

[0015] The subtitle text is displayed on the target video frame based on the time information of the speech signal.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0017] processor;

[0018] a memory for storing instructions executable by the processor;

[0019] The processor is configured to execute the instructions to implement the subtitle display method provided in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the subtitle display method provided in the first aspect.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the subtitle display method provided in the first aspect.

[0022] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0023] The audio stream and video frame sequence are separated from the video file. The audio features extracted from the audio stream can objectively and accurately reflect the voice signals and non-voice signals in the audio stream. The image features extracted from each video frame can reflect the visual changes of the video frame. Then, by combining the image features of each video frame and the audio features of the audio stream, the target video frame synchronized with the voice signal in the video frame sequence can be accurately identified. Furthermore, based on the voice signal corresponding to the target video frame, the subtitle text corresponding to the target video frame is generated, and based on the time information of the voice signal, the subtitle text is displayed on the target video frame, so that the subtitle text can be synchronized with the target video frame, ensuring that the subtitle text matches the context of the video screen. The entire process does not require human intervention, thereby improving the efficiency and accuracy of subtitle generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A flowchart of a subtitle display method provided for an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of a subtitle display device provided for one embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the present application will be clearly and completely described in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are an "or" relationship.

[0029] The subtitle display method provided in the embodiment of the present application can be executed by an electronic device, specifically by a processor of the electronic device. The electronic device referred to herein may include a terminal device, such as but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart home appliance, a smart watch, a vehicle terminal, an aircraft, etc.; or, the electronic device may also include a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0030] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0031] Please refer to Figure 1 , is a flow chart of a subtitle display method provided by an embodiment of the present application, the method comprising the following steps:

[0032] S102, separating the audio stream and the video frame sequence from the video file.

[0033] The file receiving module built into the electronic device supports receiving video files in various formats through a network interface or a user upload interface, such as but not limited to AVI, MP4, MKV, etc. After receiving the video file, the video processing software module (FFmpeg) can be used to decode the video file and separate the video frame and audio stream, wherein the separation process can be expressed as the following formula (1):

[0034] V frams , A stream =FFmpcg_dccpodc(V file ) (1)

[0035] Among them, V frams Represents a video frame sequence, which contains multiple video frames arranged in time order. stream Indicates the audio stream, V file Represents a video file, and FFmpcg_dccpodc represents decoding and separation operations.

[0036] Optionally, after obtaining the video frame sequence and the audio stream, the video frame sequence may be preprocessed, which may include, but is not limited to, adjusting the resolution of the video frame sequence, adjusting the pixel value of the video frame sequence, etc.

[0037] In order to reduce computational complexity and optimize subsequent processing steps, the video frame sequence needs to be adjusted to a standard resolution. Resolution adjustment can be achieved through downsampling or upsampling. The specific function can be expressed as the following formula (2):

[0038] [V' frames =rcsize(V frame ,W target ,H target ] (2)

[0039] Among them, V' frames Represents the video frame sequence after the resolution is adjusted, rcsize represents the operation of adjusting the resolution, V frame represents the video frame sequence before adjusting the resolution, W target Indicates the target width, H target Indicates the target altitude.

[0040] The pixel value adjustment can be achieved by normalizing the video frame sequence to improve the performance of subsequent steps. For example, the pixel value of the video frame sequence can be adjusted to the interval [0, 1] by the following formula (3).

[0041] [V”frames ={[V' frames -min(V' frames )] / [max(V' frames )-min(V' frames )]}] (3)

[0042] Among them, V frames Represents the video frame sequence after adjusting the pixel value, min(V' frames ) represents the video frame with the smallest pixel value in the video frame sequence after adjusting the resolution, max(V' frames ) represents the video frame with the largest pixel value in the video frame sequence after the resolution is adjusted.

[0043] Optionally, after obtaining the video frame sequence and the audio stream, the audio stream may be preprocessed, which may specifically include but is not limited to: noise reduction, resampling, etc.

[0044] As an example, in order to improve the accuracy of speech recognition, the audio stream can be denoised by spectral subtraction, which can be specifically expressed as the following formula (4):

[0045] [A′ stream =Spectral_Subtraction(A stream ,Noisc_Profilc)] (4)

[0046] Among them, A' stream Represents the audio stream after noise reduction, Spectral_Subtraction represents the use of spectral subtraction for noise reduction, A stream Represents the noise model extracted from the audio stream, and Noisc_Profilc represents the audio stream before noise reduction.

[0047] As an example, in order to be compatible with subsequent processing steps, the audio stream may be resampled to a specific sampling rate, which can be specifically expressed as shown in the following formula (5):

[0048] [A” stream =rcsamplc(A' stream ,ftarget)] (5)

[0049] Among them, A” stream Represents the resampled audio stream, rcsamplc represents resampling, A' stream Represents the audio stream before resampling, and ftarget represents the target sampling rate.

[0050] S104, determining a target video frame in the video frame sequence that is synchronized with the speech signal in the audio stream based on the audio features of the audio stream and the image features of each video frame in the video frame sequence.

[0051] Each video frame has a corresponding image feature. The image features of the video frame may include, for example, but are not limited to, the color histogram, edge features, key point features, spatiotemporal features (such as optical flow and motion descriptors) of the video frame. The image features of the video frame may be extracted by various appropriate feature extraction methods, such as color histogram, edge detection, key point detection and other feature extraction methods, which are not limited in the embodiments of the present application.

[0052] As a method of representing the color distribution in an image, the color histogram can be used to extract the image features of a video frame. As an example, the color histogram of a video frame can be expressed as Among them, H i (c) represents the count of color c in the histogram, c p represents the color of pixel p, δ is an indicator function, when c p Its value is 1 when =c, otherwise it is 0.

[0053] Edge detection aims to identify obvious edges in video frames. Algorithms such as the Canny edge detector can be used to detect edges in video frames. Specifically, a Gaussian filter is used to smooth the video frame to reduce noise; the gradient amplitude and direction of the video frame are calculated; non-maximum suppression is performed on the video frame; and the double threshold method is used to detect and connect edges in the video frame to obtain the edge features of the video frame.

[0054] Key point detection is to identify specific points in video frames. These specific points are identifiable and trackable in multiple video frames. It can be achieved through at least one of the following key point detection algorithms: Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF) or ORB (Oriented FAST and Rotated BRIEF), etc. can all be used to detect key points and extract their feature descriptors to obtain the key point features of the video.

[0055] Specifically, using the SIFT algorithm to detect key points in a video frame may include the following operations:

[0056] (1) Scale space extrema detection: The SIFT algorithm starts by looking for potential key points in the scale space, which are local maxima or minima in the video frame space at all scales. The scale space is constructed by applying Gaussian blur of different scales (standard deviation size) to the video frame. For each pixel, the SIFT algorithm compares the value of the pixel at different scales with the values ​​of its neighboring pixels to find the extrema.

[0057] (2) Key point location: Among the potential key points detected in the first step, some points are caused by noise or edge response in the video frame. To improve stability, the SIFT algorithm uses a fitting model to more accurately locate key points and remove points with weak response and located on the edge.

[0058] (3) Direction assignment: Each keypoint is assigned one or more directions based on the histogram of its local image gradient direction. This step is to make the feature descriptor invariant to video frame rotation.

[0059] (4) Key point descriptor generation: In the area around each key point, a local gradient direction histogram is constructed, and these histograms are combined into the feature descriptor of the point. The descriptor contains not only the location and scale information of the key point, but also the distribution information of the gradient around the point, making it invariant to changes in illumination and 3D perspective.

[0060] (5) Descriptor matching: Once the descriptors of the keypoints are created, they can be compared and matched with keypoints in other video frames. Matching is usually done by finding the nearest neighbors, such as using the Euclidean distance metric.

[0061] The audio features of the audio stream may include, for example, but are not limited to, the frequency, rhythm features (such as the number of beats per minute, etc.), intensity features, etc. of the audio stream. In the above S104, the audio features of the audio stream may be extracted from the audio stream using various appropriate feature extraction methods, which is not limited in the present embodiment.

[0062] As an example, by performing Fast Fourier Transform (FFT) on the audio stream, the frequency of the audio stream, that is, F freq =FFT(A clean ), where F freq Indicates the frequency of the audio stream, A clean represents audio stream, and FFT represents Fast Fourier Transform.

[0063] As an example, an autocorrelation function (ACF) or a peak detection algorithm is used to identify repetitive patterns in an audio stream, thereby extracting the rhythmic features of the audio stream. Specifically, first, the audio stream is divided into short-time frames, and for each short-time frame, a window function (such as a Hamming window) is used to reduce the discontinuity of the frame boundary; then, in order to strengthen the rhythmic features, each short-time frame is processed to emphasize the beats within a specific frequency range, or the beat information can be highlighted through differentiation and rectification, wherein the specific frequency range is generally a low-frequency range, because drum beats and other rhythmic features usually appear in this range; further, an autocorrelation function is used to evaluate the similarity of each short-time frame with itself at different delays, that is, Among them, R(τ) represents the autocorrelation value with a delay of τ, x[n] represents the sampling value of the short-time frame, and N represents the length of the short-time frame; finally, the periodicity in the audio stream can be found through this similarity, and this periodicity is usually related to the rhythm of the audio stream.

[0064] Further, the autocorrelation function has peaks at delays corresponding to the beat period. By detecting these peaks, the beat period of the audio stream can be estimated. Specifically, the local maximum of the ACF is detected, and a suitable threshold is selected to remove smaller peaks, which are noise or non-beat-related periodicity.

[0065] Furthermore, once the peak value of ACF is obtained, it can be converted into a beat time interval. Since ACF gives periodicity, it is necessary to convert the beat period into a time interval through an inverse transformation. Finally, the detected beat time interval is post-processed to improve stability and accuracy, including dynamic threshold adjustment, beat threshold smoothing, etc. to optimize the beat time interval and obtain the rhythm characteristics of the audio stream.

[0066] An audio stream usually contains voice signals and non-voice signals. In order to synchronize subtitles with video files, it is usually necessary to display corresponding subtitle text on the target video frame corresponding to the voice signal. In the embodiment of the present application, the target video frame can be determined by various appropriate methods.

[0067] In one embodiment, the above S104 includes the following steps:

[0068] S141, based on the image features of each video frame in the video frame sequence, determine candidate video frames in which the mouth shape of the person included in the video frame sequence has changed.

[0069] Since the video frames in which the mouth shape of the person is changed usually correspond to speech signals, by selecting such video frames from the video frame sequence as candidate video frames, reliable data support can be provided for the determination of subsequent target video frames.

[0070] The image features of the video frames reflect the visual changes of the video frames. By analyzing the image features of each video frame, it can be determined whether the mouth shape of the portrait in each video frame has changed.

[0071] As an example, the image features of each video frame are input into a lip shape recognition model to obtain the lip shape of the portrait in each video frame; the lip shapes of the portrait in different video frames in the video frame sequence are compared to determine the video frame in which the lip shape of the portrait changes as a candidate video frame.

[0072] The lip shape recognition model is a pre-trained deep learning model. The lip shape recognition model is trained based on a sample video frame sequence containing a human figure and a lip shape label corresponding to each sample video frame. The lip shape label is used to represent the lip shape of the human figure in the sample video frame, such as open mouth, closed mouth, vowels, consonants, and other different lip shapes. Therefore, the lip shape recognition model can recognize different positions and shapes of lip shapes during the training process.

[0073] More specifically, key point features, such as mouth features, are extracted from each sample video frame, and a lip shape recognition model is trained by combining key points and other features (such as color histograms or edge features) using deep learning or other machine learning techniques; then, the key point features of the video frame to be recognized are input into the trained lip shape recognition model to recognize the lip shape of the person in the video frame; finally, the lip shape of the person in the video frame is compared with the lip shapes of the person in other video frames. If they are inconsistent, it is determined that the lip shape of the person in the video frame has changed; otherwise, it is determined that the lip shape of the person in the video frame has not changed.

[0074] As another example, optical flow and shape predictor techniques can also be used to identify whether the mouth shape of a person in a video frame has changed. The optical flow method estimates motion vectors by analyzing the motion patterns of pixels between consecutive video frames, and these motion vectors can reveal the movement of the mouth. Shape predictors, such as the shape predictor in the dlib library, can identify and track facial feature points to identify whether the mouth shape of a person in a video frame has changed.

[0075] S142, based on the image features and audio features of the candidate video frame, determining whether the lip shape of the person in the candidate video frame is synchronized with the speech signal corresponding to the candidate video.

[0076] As an example, the image features and audio features of the candidate video frame are fused to obtain the fused features; the fused features are input into the synchronization detection model to obtain the detection results of the candidate video frame, and the detection results indicate whether the lip shape of the portrait in the candidate video frame is synchronized with the speech signal corresponding to the candidate video.

[0077] The synchronization detection model is trained based on sample fusion features and labels of sample video frames. The sample fusion features are obtained by fusing the image features of the sample video frames and the audio features of the sample audio stream corresponding to the sample video frames. The labels indicate whether the mouth shape of the portrait in the sample video frame is synchronized with the sample speech signal corresponding to the sample video frame in the sample audio stream. The synchronization detection model is a pre-trained deep learning model, such as CNN, RNN or Transformer, etc., which is not limited in the embodiments of the present application.

[0078] The fused features obtained by fusing the image features and audio features of the candidate video frames imply the emotions, events, etc. expressed by the candidate video frames, which helps to accurately identify whether the lip shape of the person in the candidate video frames is synchronized with the speech signal corresponding to the candidate video frames.

[0079] The fusion of image features and audio features can be achieved through various feature fusion technologies, which are not limited in the embodiments of the present application.

[0080] As an example, assuming that the audio feature is A and the image feature of the candidate video frame is V, the fusion result can be expressed as: [F{fusion}] = [A, V], where [A, V] means connecting the audio feature A and the image feature V of the candidate video frame to form a larger fusion feature.

[0081] As another example, multimodal fusion technology can be used, such as weighting features through an attention mechanism or using a multimodal deep learning architecture to fuse audio features and image features of candidate video frames, thereby effectively utilizing the complementary properties of multimodal data and improving classification accuracy.

[0082] The fused features are input into the synchronous detection model for classification to obtain the detection results of the candidate video frames.

[0083] S143: If the lip shape of the person in the candidate video frame is synchronized with the voice signal corresponding to the candidate video, the candidate video frame is determined as the target video frame.

[0084] In this embodiment, considering that identifying the target video frame corresponding to the voice signal based solely on lip shape changes is prone to misjudgment, and the lip shape of the person in the video frame corresponding to the voice signal is usually synchronized with the voice signal, based on this, the candidate video frame corresponding to the voice frame is first identified using lip shape changes, and then the image features of the candidate video frame and the audio features of the audio stream are combined to further identify whether the lip shape of the person in the candidate video frame is synchronized with the voice signal corresponding to the candidate video frame. This can accurately identify the target video frame with greater robustness.

[0085] In another embodiment, the above S104 includes the following steps: performing activity detection on the audio stream based on audio features to obtain a voice signal in the audio stream; further, based on the time information of the voice signal, determining a candidate video frame corresponding to the voice signal from a video frame sequence; further, based on the image features of the candidate video frame, identifying whether the mouth shape of the portrait in the candidate video frame has changed; if changed, determining the candidate video frame as the target video frame.

[0086] Voice Activity Detection (VAD) is designed to detect whether there is speech in an audio stream. It analyzes the audio stream to determine whether the audio frames in a certain period of time contain valid speech or invalid speech (such as noise or silence). Audio frames that contain valid speech (i.e., speech signals) in the audio stream are called speech frames, and audio frames that contain invalid speech (i.e., non-speech signals) in the audio stream are called non-speech frames.

[0087] As an example, the amplitude of the signal in the audio stream is compared with a preset amplitude threshold, and the signal exceeding the amplitude threshold is regarded as a speech signal, thereby obtaining a speech frame containing the speech signal.

[0088] As another example, each audio frame in the audio stream is input into an activity detection model to identify whether each audio frame is a speech frame. The activity detection model is a pre-trained deep learning model, such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer. The activity detection model is trained based on sample audio frames and their labels, and the label of the sample audio frame is used to indicate whether the sample audio frame is a speech frame.

[0089] The above describes some implementation methods of the above S104. Of course, it should be understood that the above S104 can also be implemented in other ways, which are not limited in the present embodiment.

[0090] S106, generating a subtitle text corresponding to the target video frame based on the speech signal corresponding to the target video frame.

[0091] The subtitle text is used to describe the content of the target video frame. The above S106 can be implemented in various appropriate ways.

[0092] In one embodiment, the above S106 includes the following steps: performing speech recognition on the speech frame corresponding to the target video frame, such as performing speech recognition on the speech frame using automatic speech recognition technology (Automatic Speech Recognition, ASR), and using the recognized text as the subtitle text corresponding to the target video frame.

[0093] In another embodiment, the above S106 includes the following steps:

[0094] S161, performing speech recognition on a speech signal corresponding to a target video frame to obtain a first text.

[0095] Speech recognition of speech frames can be achieved using various ASR technologies in the art, and the embodiments of the present application are not limited to this.

[0096] S162: Generate a plurality of second texts based on the keywords in the first text and the first language model, and determine a score for each second text.

[0097] Keywords refer to words that can reflect the semantics of a text. In the embodiment of the present application, keywords in a text can be extracted by various appropriate methods. Specifically, the text is segmented to obtain a word set, and the word set is subjected to denoising (such as noise, irrelevant filler words, etc.), stemming, lemmatization, and stop words, etc., to reduce the size of the word set and improve the efficiency and accuracy of subsequent processing; then, keywords are determined from the word set through a bag of words model (Bag of Words, BoW), term frequency-inverse document frequency (Term Frequency-Inverse Document Frequency, TF-IDF), a named entity recognition model, etc.

[0098] As an example, for each word in the word set, the term frequency (TF) and inverse document frequency (IDF) of the word are calculated respectively; then, based on the term frequency and inverse document frequency of the word, the TF-IDF score of the word for the text is calculated, and the score indicates the importance of the word for the text, that is, the higher the TF-IDF score of the word, the more important the word is for the document; finally, the word with a TF-IDF score greater than or equal to the preset score is selected from the word set as a keyword. Among them, the term frequency of the word can be obtained by the following formula (6), and the inverse document frequency of the word can be obtained by the following formula (7).

[0099]

[0100]

[0101] Among them, TF(t,d) represents the term frequency of word t, and IDF(t) represents the inverse document frequency of word t.

[0102] As another example, the text is input into a named entity recognition (NER) model to obtain words representing named entities in the text, and these words are used as keywords. Among them, the named entity recognition model can adopt various machine learning models with named entity recognition functions in the art, such as conditional random fields (CRF) model, long short-term memory network (LSTM), LSTM-CRF model, etc., which are not limited to this embodiment of the present application. Among them, the LSTM-CRF model is a combination of LSTM and CRF models, which first captures the text information before and after the text through LSTM, and then the CRF model uses these context information to identify whether each word represents a named entity (such as a name, a place name, an organization name, etc.). Words and the named entities they represent can assist in determining the subtitle text, such as giving priority to displaying names, proper names or keywords of a certain category, making the subtitle text richer and more informative.

[0103] After obtaining the keywords, the text generation capability of the first language model may be used to generate text based on the keywords to obtain a plurality of second texts.

[0104] As an example, the first language model can be a sequence-to-sequence model, such as RNN, LSTM, Transformer, etc. The keyword is input into the first language model, or the keyword is combined with its context information (such as text, the mouth shape of a person in a target video frame, etc.) and then input into the first language model. The first language model encodes the input into a vector representation of a fixed length, and then decodes the vector representation to generate multiple second texts related to the keyword.

[0105] As another example, the first language model may be a large language model (LLM), such as the GPT series or a self-developed large language model, etc. The prompt word and the keyword are concatenated and input into the first language model, and the first language model generates a plurality of second texts related to the keyword according to the instruction of the prompt word.

[0106] The above describes some implementation methods of generating multiple second texts. Of course, it should be understood that multiple second texts can also be implemented in other ways, and the embodiments of the present application are not limited to this.

[0107] After obtaining a plurality of second texts, the score of each second text may be determined in various appropriate ways. The score of the second text is used to indicate the accuracy of the second text.

[0108] As an example, for each second text, for each second text, obtain the conditional probability predicted by the first language model for each word in the second text; determine the perplexity of the second text based on the product of the conditional probabilities of each word in the second text; and determine the score of the second text based on the perplexity of the second text. Specifically, the perplexity of the second text is obtained by the following formula (8):

[0109]

[0110] Among them, Perplexity(W) represents the perplexity of the second text W, represents the first word contained in the second text W, represents the second word contained in the second text W, The Nth word contained in the second text W, P represents the conditional probability predicted by the first language model, and N represents the number of words contained in the second text W.

[0111] The perplexity of the second text indicates the prediction ability of the first language model for the second text, that is, the lower the perplexity, the stronger the prediction ability of the first language model for the second text. Further, for each second text, the inverse of the perplexity of the second text can be determined as the score of the second text, and the score can reflect the accuracy of the second text.

[0112] As another example, for each second text, a second language model is used to predict the index value of the second text in multiple semantic-related indicators; the index values ​​of the second text in multiple semantic-related indicators are weighted and summed to obtain the score of the second text. Among them, multiple speech-related indicators may, for example, include but are not limited to: semantic relevance, grammatical correctness, comprehensibility, etc. Semantic relevance indicates the correlation between the second text and the content of the target video frame, grammatical correctness indicates whether the grammar of the second text is correct, and comprehensibility indicates whether the second text is easy for the audience to understand. The weight corresponding to each semantic-related indicator can be set according to actual needs, and the embodiment of the present application is not limited to this.

[0113] Specifically, the first language model can be a large language model, such as the GPT series or a self-developed large language model. The second text and the prompt word for indicating the score are concatenated and input into the second language model, and the first language model predicts the index values ​​of the second text in multiple indicators according to the indication of the prompt word; the index values ​​of the second text in multiple indicators are weighted and summed to obtain the score of the second text.

[0114] S163: Select the second text with the highest score from the multiple second texts as the subtitle text corresponding to the target video frame.

[0115] Specifically, the second text with the highest score can be selected as the subtitle text corresponding to the target video frame by maximizing the score function, that is, Best Caption = arg max(Score(Caption)), where Score(Caption) represents the score of the second text, which represents the subtitle text corresponding to the target video frame.

[0116] In the above implementation, since the keywords in the text corresponding to the target video frame can accurately reflect the semantics of the text, the second text generated based on the keywords is more consistent with the semantics of the text; on this basis, by scoring the second text, the accuracy of the second text can be more objectively and accurately reflected, and then the second text with the highest score is selected as the subtitle text corresponding to the target video frame, so that the subtitle text is more consistent with the content of the target video frame.

[0117] The above describes some implementation methods of S106. Of course, it should be understood that the above S106 can also be implemented in other ways, which are not limited in the present embodiment.

[0118] S108, displaying subtitle text on the target video frame based on the time information of the speech signal.

[0119] The time information of the speech signal includes the start timestamp and the end timestamp of the speech signal in the audio stream.

[0120] In order to ensure the synchronization between the subtitle text and the speech signal corresponding to the target video frame, in one embodiment, the time period between the start timestamp and the end timestamp is determined as the display time period of the subtitle text; within the display time period, the subtitle text is displayed on the target video frame.

[0121] As an example, the start timestamp of the speech frame corresponding to the target video frame is 1 minute 20 seconds, and the end timestamp is 1 minute 25 seconds, then the display time period of the subtitle text is determined to be 1 minute 20 seconds to 1 minute 25 seconds, and then the subtitle text is displayed within this time period, for example, the subtitle text is displayed on the target video frame from 1 minute 20 seconds, and the subtitle text is deleted from the target video frame at 1 minute 25 seconds.

[0122] In order to avoid the subtitle text being displayed too fast or too slow and affecting the audience's reading, as another example, the subtitle text is displayed on the target video frame within the display time period, including the following steps: Step A1, obtaining the number of words contained in the subtitle text and the reference reading time of a single word; Step A2, determining the continuous display time of the subtitle text based on the product of the number of words and the reference reading time; Step A3, displaying the subtitle text on the target video frame within the display time period based on the continuous display time.

[0123] Specifically, in step A2, the product of the number of words and the reference reading time may be determined as the continuous display time of the subtitle text.

[0124] Alternatively, the reaction time of the readers targeted by the video file may be obtained, the product between the number of words and the reference reading time may be determined, and the sum of the product and the reaction time may be determined as the continuous display time of the subtitle text.

[0125] For example, the continuous display duration of the subtitle text can be determined by the following formula (9):

[0126] T d =(N w ×T w )+T e (9)

[0127] Among them, T d Indicates the duration of subtitle text display, N w Indicates the number of words contained in the subtitle text, T w Indicates the reference reading time of a single word, T e Indicates the duration of readers' response to the video file.

[0128] The reference reading time of a single word is used to indicate the time it takes a reader to read a word, which can be determined according to the reader's reading speed. Specifically, an adult's reading speed is about 200 to 300 words per minute, so the reference reading time of a single word is Where represents the number of words read per minute.

[0129] The reader's reaction time is obtained through psychological research data, which is generally 200 to 300 milliseconds.

[0130] In order to enhance the display effect of subtitles to ensure the best viewing experience, as another example, subtitle text is displayed on a target video frame within a display time period, including the following steps: step B1, obtaining display parameters of the target video frame; step B2, determining a target display style of the subtitle text based on the display parameters; step B3, displaying the subtitle text on the target video frame within the display time period based on the target display style.

[0131] Specifically, in step B1, the display parameters may include, for example, but are not limited to: a height value of a screen for displaying a target video frame, a color and brightness of the target video frame, and the like.

[0132] In step B2, the style of the subtitle text may include, but is not limited to: font size, color, brightness, etc. of the subtitle text.

[0133] Exemplarily, the display parameter includes a height value of a screen for displaying a target video frame, and the target display style includes a target font size. The target font size can be determined by: determining a ratio between the height value of the screen and a preset scale factor; and determining a target font size of the subtitle text based on the ratio.

[0134] For example, the target font size is determined by the following formula (10):

[0135]

[0136] Wherein, S represents the target font size, H represents the height value of the screen, and N represents the preset scale factor. The preset scale factor is a constant determined based on readability research, that is, the scale factor between the height value of the screen and the font size, which depends on multiple factors such as viewing distance, screen resolution, audience vision, and reading comfort. For a general TV screen, the font size is determined based on a certain ratio of the screen height value. For example, the font size can be 1 / 30 to 1 / 20 of the screen height value, which can ensure that the subtitles are clear and readable at a standard viewing distance, so the preset scale factor can range from 20 to 30. For example, if N = 30, for a TV with a screen height value of 1080 pixels, the target font size S = 1080 / 30 = 36 pixels; if N = 20, using a TV with the same height screen, the target font size S = 1080 / 20 = 54 pixels.

[0137] In actual applications, the optimal font size needs to be determined based on the actual situation and the specific needs of users. The optimal value of N will be different for different viewing environments and subtitle usage scenarios (such as mobile phones, tablets, computers, and TVs). Further fine-tuning the value of N through user testing and feedback can ensure the best reading experience.

[0138] Exemplarily, the display parameter includes the brightness of the target video frame, and the target display style includes the target brightness of the subtitle text. The target brightness can be determined according to a preset brightness contrast and the brightness of the target video frame, specifically determined by the following formula (11):

[0139]

[0140] Among them, C represents the brightness contrast, Y bg Indicates the brightness of the target video frame, Y text Indicates the brightness of the subtitle text.

[0141] Exemplarily, the display parameter includes the color of the target video frame, and the target display style includes the target color of the subtitle text. The target color can be determined according to a preset color contrast and the color of the target video frame, specifically determined by the following formula (12):

[0142]

[0143] Among them, C represents the color contrast, C text Represents the RGB value of the color of the target video frame, C bg In order to achieve adaptive adjustment of the display style of the subtitle text, as another example, the subtitle text is displayed on the target video frame during the display time period, including the following steps:

[0144] Step C1, generating a particle swarm and an evaluation function corresponding to the particle swarm.

[0145] The particle group includes multiple particles. Each particle has a corresponding speed and position, and the position of each particle corresponds to a display style of the subtitle text.

[0146] The evaluation function is used to evaluate the fitness of each particle, and the fitness represents the display effect of the display style corresponding to the position of the particle. As an example, the evaluation function includes a first contrast function and a second contrast function. Among them, the first contrast function is used to evaluate the contrast between the brightness corresponding to the position of each particle and the brightness of the target video frame, such as the first contrast function can use the above formula (11). The second contrast function is used to evaluate the contrast between the color corresponding to the position of each particle and the color of the target video frame, such as the second contrast function can use the above formula (12). As another example, the evaluation function can also combine the content of the subtitle text and the context information of the target video frame to evaluate the fitness of each particle, that is, the evaluation function can be expressed as f(style, content, context), wherein style represents the display style of the subtitle text, content represents the content of the subtitle text, and context represents the context information of the target video frame.

[0147] In step C2, the fitness of each particle is determined based on the position of each particle and the evaluation function.

[0148] Specifically, the display style corresponding to the position of each particle is used as the input parameter of the evaluation function to calculate the fitness of each particle.

[0149] Step C3, based on the fitness of each particle, determine the historical optimal position of each particle and the global optimal position of the particle swarm.

[0150] Specifically, for each particle, the fitness of the particle at the current position is compared with the fitness of the particle at the historical position, and the position with the best display effect represented by the fitness is determined as the historical optimal position of the particle; if the particle does not have a historical position, the current position of the particle is determined as the historical optimal position of the particle.

[0151] For a particle swarm, the fitness of all particles in the particle swarm at the current position is compared, and the position with the best display effect represented by the fitness is determined as the global optimal position of the particle swarm.

[0152] Step C4, displaying the subtitle text on the target video frame based on the display style corresponding to the global optimal position.

[0153] Step C5, based on the historical optimal position of each particle, the global optimal position and the speed of each particle, the position of each particle is updated.

[0154] Specifically, the position of each particle can be updated according to the historical optimal position of the particle, the global optimal position, the velocity, and the corresponding relationship between the positions.

[0155] Repeat the above steps C2 to C5 until the display end condition within the display time period is met.

[0156] The end display condition can be set according to actual needs, such as the current timestamp reaches the end time period of the display time period or the display duration of the subtitle text reaches the continuous display duration, etc., which is not limited in this embodiment of the present application.

[0157] During the display period, the above steps C2 to C5 are repeatedly executed to achieve dynamic adaptive adjustment of the display parameters of the subtitle text, thereby ensuring the best display effect of the subtitle text.

[0158] One or more embodiments of the present application provide a subtitle display method, which separates an audio stream and a video frame sequence from a video file. The audio features extracted from the audio stream can objectively and accurately reflect the voice signals and non-voice signals in the audio stream. The image features extracted from each video frame can reflect the visual changes of the video frame. Then, by combining the image features of each video frame and the audio features of the audio stream, the target video frame synchronized with the voice signal in the video frame sequence can be accurately identified. Furthermore, based on the voice signal corresponding to the target video frame, a subtitle text corresponding to the target video frame is generated, and based on the time information of the voice signal, the subtitle text is displayed on the target video frame, so that the subtitle text can be synchronized with the target video frame, ensuring that the subtitle text matches the context of the video screen. Moreover, the entire process does not require manual intervention, thereby improving the efficiency and accuracy of subtitle generation.

[0159] The subtitle display method provided in the embodiment of the present application is applicable to various application scenarios and implementation environments. As an example, the subtitle display method provided in the embodiment of the present application is applicable to a cloud terminal system, and provides a scalable and efficient subtitle generation and display service by using the elastic resources of cloud computing, meeting the needs of different users and coping with large-scale video processing needs.

[0160] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0161] Based on the same inventive concept, the present application also provides a subtitle display device. Figure 2 , is a schematic structural diagram of a subtitle display device 200 provided in one embodiment of the present application, the device 200 comprises:

[0162] A separation module 210, used to separate an audio stream and a video frame sequence from a video file;

[0163] A first determination module 220, configured to determine a target video frame in the video frame sequence that is synchronized with a speech signal in the audio stream based on an audio feature of the audio stream and an image feature of each video frame in the video frame sequence;

[0164] A first generating module 230, configured to generate a subtitle text corresponding to the target video frame based on a speech signal corresponding to the target video frame;

[0165] The display module 240 is configured to display the subtitle text on the target video frame based on the time information of the speech signal.

[0166] In another embodiment, the first determining module is used to:

[0167] Based on the image features of each video frame in the video frame sequence, determining a candidate video frame in which the mouth shape of the person included in the video frame sequence changes from the video frame sequence;

[0168] Based on the image features and the audio features of the candidate video frame, determining whether the lip shape of the person in the candidate video frame is synchronized with the speech signal corresponding to the candidate video;

[0169] If the lip shape of the person in the candidate video frame is synchronized with the voice signal corresponding to the candidate video, the candidate video frame is determined as the target video frame.

[0170] In another embodiment, the first determination module performs the following steps when determining, from the video frame sequence, a candidate video frame in which the mouth shape of the person portrait contained therein changes based on the image features of each video frame in the video frame sequence:

[0171] The image features of each video frame are input into the lip shape recognition model to obtain the lip shape of the person in each video frame;

[0172] The mouth shapes of the portrait in different video frames in the video frame sequence are compared to determine the video frame in which the mouth shape of the portrait changes as the candidate video frame.

[0173] In another embodiment, when the first determination module determines whether the lip shape of the person in the candidate video frame is synchronized with the voice signal corresponding to the candidate video based on the image feature and the audio feature of the candidate video frame, the first determination module performs the following steps:

[0174] Fusing the image feature and the audio feature of the candidate video frame to obtain a fused feature;

[0175] Inputting the fusion feature into a synchronization detection model to obtain a detection result of the candidate video frame, wherein the detection result indicates whether the lip shape of the person in the candidate video frame is synchronized with the speech signal corresponding to the candidate video;

[0176] Among them, the synchronization detection model is trained based on sample fusion features and labels of sample video frames, the sample fusion features are obtained by fusing image features of the sample video frames and audio features of the sample audio stream corresponding to the sample video frames, and the labels indicate whether the lip shape of the portrait in the sample video frame is synchronized with the sample voice signal corresponding to the sample video frame in the sample audio stream.

[0177] In another embodiment, the first generating module is used to:

[0178] Performing speech recognition on the speech signal corresponding to the target video frame to obtain a first text;

[0179] Based on the keywords in the first text and the first language model, generate a plurality of second texts, and determine a score for each second text;

[0180] A second text with the highest score is selected from the multiple second texts as the subtitle text corresponding to the target video frame.

[0181] In another embodiment, the first generating module performs the following steps when determining the score of each second text:

[0182] For each second text, obtaining the conditional probability predicted by the first language model for each word in the second text;

[0183] Determining the perplexity of the second text based on the product of the conditional probabilities of each word in the second text;

[0184] Based on the perplexity of the second text, a score of the second text is determined.

[0185] In another embodiment, the first generating module performs the following steps when determining the score of each second text:

[0186] For each second text, predicting the index values ​​of the second text in multiple semantic related indexes by using a second language model;

[0187] The index values ​​of the multiple semantic related indexes are weightedly summed to obtain the score of the second text.

[0188] In another embodiment, the time information includes a start timestamp and an end timestamp;

[0189] The display module is used for:

[0190] Determine the time period between the start timestamp and the end timestamp as the display time period of the subtitle text;

[0191] During the display time period, the subtitle text is displayed on the target video frame.

[0192] In another embodiment, the display module performs the following steps when displaying the subtitle text on the target video frame within the display time period:

[0193] Obtaining the number of words contained in the subtitle text and the reference reading time of each word;

[0194] Determining the continuous display duration of the subtitle text based on the product of the number of words and the reference reading duration;

[0195] During the display time period, the subtitle text is displayed on the target video frame based on the continuous display duration.

[0196] In another embodiment, when determining the continuous display duration of the subtitle text based on the product of the number of words and the reference reading duration, the display module performs the following steps:

[0197] Obtaining the reaction time of readers targeted by the video file;

[0198] determining the product between the number of words and the reference reading time;

[0199] The sum of the product and the reaction time is determined as the continuous display time of the subtitle text.

[0200] In another embodiment, the display module performs the following steps when displaying the subtitle text on the target video frame within the display time period:

[0201] Acquire display parameters of the target video frame;

[0202] Based on the display parameters, determining a target display style of the subtitle text;

[0203] During the display time period, the subtitle text is displayed on the target video frame based on the target display style.

[0204] In another embodiment, the display parameter includes a height value of a screen for displaying the target video frame, and the display style includes a target font size of the subtitle text:

[0205] The target font size is determined as follows:

[0206] Determine a ratio between a height value of the screen and a preset scale factor;

[0207] Based on the ratio, a target font size of the subtitle text is determined.

[0208] In another embodiment, the display module performs the following steps when displaying the subtitle text on the target video frame within the display time period:

[0209] Generate a particle swarm and an evaluation function corresponding to the particle swarm, the particle swarm includes a plurality of particles, each particle has a corresponding speed and position, the position of each particle corresponds to a display style of the subtitle text, the evaluation function is used to evaluate the fitness of each particle, and the fitness represents the display effect of the display style corresponding to the position of the particle;

[0210] Repeat the following steps until the display end condition is met:

[0211] Determining the fitness of each particle based on the position of each particle and the evaluation function;

[0212] Based on the fitness of each particle, determining the historical optimal position of each particle and the global optimal position of the particle swarm;

[0213] Based on the display style corresponding to the global optimal position, displaying the subtitle text on the target video frame;

[0214] The position of each particle is updated based on the historical optimal position of each particle, the global optimal position and the speed of each particle.

[0215] In another embodiment, the display style includes the color and brightness of the subtitle text;

[0216] The evaluation function includes a first contrast function and a second contrast function, wherein the first contrast function is used to evaluate the contrast between the brightness corresponding to the position of each particle and the brightness of the target video frame, and the second contrast function represents the difference between the color corresponding to the position of each particle and the color of the target video frame.

[0217] Obviously, the subtitle display device 200 provided in the embodiment of the present application can be used as the above Figure 1 The execution body of the subtitle display method shown in the figure can realize the subtitle display device in Figure 1 The functions realized are as follows. Since the principles are the same, they will not be described in detail.

[0218] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0219] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0220] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0221] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a subtitle display device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0222] Separate audio stream and video frame sequence from video files;

[0223] Determine, based on the audio features of the audio stream and the image features of each video frame in the video frame sequence, a target video frame in the video frame sequence that is synchronized with the voice signal in the audio stream;

[0224] Based on the speech signal corresponding to the target video frame, generating a subtitle text corresponding to the target video frame;

[0225] The subtitle text is displayed on the target video frame based on the time information of the speech signal.

[0226] The above application Figure 1 The method performed by the subtitle display device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in software form. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0227] The electronic device may also perform Figure 1 The method and the subtitle display device are implemented in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0228] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0229] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations:

[0230] Separate audio stream and video frame sequence from video files;

[0231] Determine, based on the audio features of the audio stream and the image features of each video frame in the video frame sequence, a target video frame in the video frame sequence that is synchronized with the voice signal in the audio stream;

[0232] Based on the speech signal corresponding to the target video frame, generating a subtitle text corresponding to the target video frame;

[0233] The subtitle text is displayed on the target video frame based on the time information of the speech signal.

[0234] The embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps in the subtitle display method provided in the embodiment of the present application.

[0235] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0236] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0237] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0238] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0239] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A subtitle display method, characterized in that: include: Separate audio stream and video frame sequence from video files; Determine, based on the audio features of the audio stream and the image features of each video frame in the video frame sequence, a target video frame in the video frame sequence that is synchronized with the voice signal in the audio stream; Based on the speech signal corresponding to the target video frame, generating a subtitle text corresponding to the target video frame; The subtitle text is displayed on the target video frame based on the time information of the speech signal.

2. The method according to claim 1, characterized in that The step of determining a target video frame in the video frame sequence that is synchronized with a speech signal in the audio stream based on the audio feature of the audio stream and the image feature of each video frame in the video frame sequence comprises: Based on the image features of each video frame in the video frame sequence, determining a candidate video frame in which the mouth shape of the person included in the video frame sequence changes from the video frame sequence; Based on the image features and the audio features of the candidate video frame, determining whether the lip shape of the person in the candidate video frame is synchronized with the speech signal corresponding to the candidate video; If the lip shape of the person in the candidate video frame is synchronized with the voice signal corresponding to the candidate video, the candidate video frame is determined as the target video frame.

3. The method according to claim 2, characterized in that The step of determining, based on the image features of each video frame in the video frame sequence, a candidate video frame in which the mouth shape of the person portrait contained in the video frame sequence changes, comprises: The image features of each video frame are input into the lip shape recognition model to obtain the lip shape of the person in each video frame; The mouth shape of the person in each video frame is compared to determine the video frame in which the mouth shape of the person in the video frame sequence changes as the candidate video frame.

4. The method according to claim 2, characterized in that: The determining, based on the image features and the audio features of the candidate video frame, whether the lip shape of the person in the candidate video frame is synchronized with the voice signal corresponding to the candidate video, comprises: Fusing the image feature and the audio feature of the candidate video frame to obtain a fused feature; Inputting the fusion feature into a synchronization detection model to obtain a detection result of the candidate video frame, wherein the detection result indicates whether the lip shape of the person in the candidate video frame is synchronized with the speech signal corresponding to the candidate video; Among them, the synchronization detection model is trained based on sample fusion features and labels of sample video frames, the sample fusion features are obtained by fusing image features of the sample video frames and audio features of the sample audio stream corresponding to the sample video frames, and the labels indicate whether the lip shape of the portrait in the sample video frame is synchronized with the sample voice signal corresponding to the sample video frame in the sample audio stream.

5. The method according to claim 1, characterized in that The step of generating a subtitle text corresponding to the target video frame based on the speech signal corresponding to the target video frame comprises: Performing speech recognition on the speech signal corresponding to the target video frame to obtain a first text; Based on the keywords in the first text and the first language model, generate a plurality of second texts, and determine a score for each second text; A second text with the highest score is selected from the multiple second texts as the subtitle text corresponding to the target video frame.

6. The method according to claim 5, characterized in that Determining the score of each second text includes: For each second text, obtaining the conditional probability predicted by the first language model for each word in the second text; Determining the perplexity of the second text based on the product of the conditional probabilities of each word in the second text; Based on the perplexity of the second text, a score of the second text is determined.

7. The method according to claim 5, characterized in that Determining the score of each second text includes: For each second text, predicting the index values ​​of the second text in multiple semantic related indexes by using a second language model; The index values ​​of the multiple semantic related indexes are weightedly summed to obtain the score of the second text.

8. The method according to claim 1, characterized in that: The time information includes a start timestamp and an end timestamp; The step of displaying the subtitle text on the target video frame based on the time information of the speech signal comprises: Determine the time period between the start timestamp and the end timestamp as the display time period of the subtitle text; During the display time period, the subtitle text is displayed on the target video frame.

9. The method according to claim 8, characterized in that The step of displaying the subtitle text on the target video frame within the display time period includes: Obtaining the number of words contained in the subtitle text and the reference reading time of each word; Determining the continuous display duration of the subtitle text based on the product of the number of words and the reference reading duration; During the display time period, the subtitle text is displayed on the target video frame based on the continuous display duration.

10. The method according to claim 9, characterized in that The step of determining the continuous display duration of the subtitle text based on the product of the number of words and the reference reading duration includes: Obtaining the reaction time of readers targeted by the video file; determining the product of the number of words and the reference reading time; The sum of the product and the reaction time is determined as the continuous display time of the subtitle text.

11. The method according to claim 8, characterized in that The step of displaying the subtitle text on the target video frame within the display time period includes: Acquire display parameters of the target video frame; Based on the display parameters, determining a target display style of the subtitle text; During the display time period, the subtitle text is displayed on the target video frame based on the target display style.

12. The method according to claim 11, characterized in that The display parameters include a height value of a screen for displaying the target video frame, and the display style includes a target font size of the subtitle text: The target font size is determined as follows: Determine a ratio between a height value of the screen and a preset scale factor; Based on the ratio, a target font size of the subtitle text is determined.

13. The method according to claim 8, characterized in that The step of displaying the subtitle text on the target video frame within the display time period includes: Generate a particle swarm and an evaluation function corresponding to the particle swarm, the particle swarm includes a plurality of particles, each particle has a corresponding speed and position, the position of each particle corresponds to a display style of the subtitle text, the evaluation function is used to evaluate the fitness of each particle, and the fitness represents the display effect of the display style corresponding to the position of the particle; Repeat the following steps until the display end condition is met: Determining the fitness of each particle based on the position of each particle and the evaluation function; Based on the fitness of each particle, determining the historical optimal position of each particle and the global optimal position of the particle swarm; Based on the display style corresponding to the global optimal position, displaying the subtitle text on the target video frame; The position of each particle is updated based on the historical optimal position of each particle, the global optimal position and the speed of each particle.

14. The method according to claim 13, characterized in that The display style includes the color and brightness of the subtitle text; The evaluation function includes a first contrast function and a second contrast function, wherein the first contrast function is used to evaluate the contrast between the brightness corresponding to the position of each particle and the brightness of the target video frame, and the second contrast function represents the difference between the color corresponding to the position of each particle and the color of the target video frame.

15. A subtitle display device, characterized in that: include: A separation module, used for separating audio stream and video frame sequence from a video file; Determine, based on the audio features of the audio stream and the image features of each video frame in the video frame sequence, a target video frame in the video frame sequence that is synchronized with the voice signal in the audio stream; Based on the speech signal corresponding to the target video frame, generating a subtitle text corresponding to the target video frame; The subtitle text is displayed on the target video frame based on the time information of the speech signal.

16. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the subtitle display method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the subtitle display method according to any one of claims 1 to 14.

18. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the subtitle display method according to any one of claims 1 to 14.

Images