A video subtitle generation method, device, equipment and storage medium

Abstract

The application discloses a video subtitle generation method and device, equipment and a storage medium, and relates to the technical field of video processing. The method comprises the following steps: decomposing a video into a video frame sequence, an audio signal stream and a speech transcription text; performing feature extraction on the video frame sequence, the audio signal stream and the speech transcription text to obtain a visual feature vector sequence, an acoustic feature representation and a semantic representation; fusing the visual feature vector sequence, the acoustic feature representation and the semantic representation to obtain a multi-modal video feature representation; performing emotion recognition and quantization on the multi-modal video feature representation to obtain an emotion vector, and obtaining an emotion intensity according to the emotion vector; identifying a speaker identity in the video and a scene type of the video; selecting a subtitle style parameter according to the emotion intensity, the speaker identity and the scene type; and rendering a subtitle according to the subtitle style parameter. The method can improve efficiency, reduce cost and improve the effect of the subtitle.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a method, apparatus, device and storage medium for generating video subtitles. Background Technology

[0002] With the rapid development of mobile internet and short video platforms, micro-dramas have become an important part of online audiovisual content. In the production process of micro-dramas, subtitles are a key factor affecting the user viewing experience and the effectiveness of content dissemination. High-quality subtitles not only help users understand the dialogue but also enhance the emotional expression through visual presentation, improving the professionalism and watchability of the video. However, current subtitle production methods suffer from low efficiency, high cost, and poor subtitle quality.

[0003] Therefore, how to solve the above-mentioned technical defects has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a video subtitle generation method, apparatus, device, and storage medium that can improve efficiency, reduce costs, and enhance subtitle quality.

[0005] To address the aforementioned technical problems, this application provides a video subtitle generation method, comprising:

[0006] The video is broken down into a sequence of video frames, an audio signal stream, and speech-to-text transcription.

[0007] Feature extraction is performed on the video frame sequence, the audio signal stream, and the speech-transcribed text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively.

[0008] The visual feature vector sequence, the acoustic feature representation, and the semantic representation are fused to obtain a multimodal video feature representation;

[0009] Emotion vectors are obtained by performing emotion recognition and quantization based on the multimodal video feature representation, and the emotion intensity is obtained based on the emotion vectors.

[0010] Identify the speaker's identity and the video scene type in the video;

[0011] Select subtitle style parameters based on the emotional intensity, the speaker's identity, and the scene type;

[0012] Render the subtitles according to the subtitle style parameters.

[0013] In some embodiments, the emotion vector includes three dimensions: pleasure, activation, and dominance; the pleasure reflects the positive or negative tendency of the emotion; the activation reflects the intensity of the emotion; and the dominance reflects the active or passive characteristics of the emotion.

[0014] In some embodiments, identifying the speaker in a video includes:

[0015] The speaker's identity in the video is identified using face detection, voice separation, and lip reading technologies.

[0016] In some embodiments, the scene type identified in the video includes:

[0017] Segment the video into semantically coherent scene units;

[0018] Each scene unit is classified by scene type.

[0019] In some embodiments, selecting subtitle style parameters based on the emotional intensity, the speaker's identity, and the scene type includes:

[0020] Select font parameters based on the intensity of the emotion and the type of the scene;

[0021] Select color parameters based on the intensity of the emotion and the speaker's identity;

[0022] Select the location parameters according to the scene type;

[0023] Select the motion effect parameters based on the intensity of the emotion and the type of the scene.

[0024] In some embodiments, rendering subtitles according to the subtitle style includes:

[0025] Synchronize and align subtitle segments with video frames based on timeline information;

[0026] Generate the corresponding subtitle layer based on the subtitle style parameters of each subtitle;

[0027] Composite the subtitle layer with the original video footage;

[0028] For subtitles that require animation, keyframe animations are generated based on the animation parameters and played according to the preset timeline during video playback.

[0029] In some embodiments, feature extraction is performed on the video frame sequence, the audio signal stream, and the speech-transcribed text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively, including:

[0030] The spatial feature representation of each frame in the video frame sequence is obtained by an image feature extractor, and the temporal relationship between frames is modeled by a long short-term memory network or a spatiotemporal convolutional network to obtain the visual feature vector sequence.

[0031] The audio signal stream is subjected to frame-by-frame windowing processing, and the Mel spectrogram and Mel frequency cepstral coefficients are extracted as the acoustic feature representation;

[0032] The transcribed text is segmented and vectorized, and a language model is used to obtain the semantic representation.

[0033] To address the aforementioned technical problems, this application also provides a video subtitle generation apparatus, comprising:

[0034] The decomposition unit is used to decompose video into a sequence of video frames, an audio signal stream, and speech-to-text.

[0035] The extraction unit is used to extract features from the video frame sequence, the audio signal stream, and the speech-transcribed text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively.

[0036] A fusion unit is used to fuse the visual feature vector sequence, the acoustic feature representation, and the semantic representation to obtain a multimodal video feature representation;

[0037] An emotion recognition unit is used to perform emotion recognition and quantization based on the multimodal video feature representation to obtain an emotion vector, and to obtain the emotion intensity based on the emotion vector;

[0038] The scene recognition unit is used to identify the speaker's identity and the scene type in the video;

[0039] The selection unit is used to select subtitle style parameters based on the emotional intensity, the speaker's identity, and the scene type.

[0040] A rendering unit is used to render subtitles according to the subtitle style parameters.

[0041] To address the aforementioned technical problems, this application also provides an electronic device, comprising:

[0042] Memory, used to store computer programs;

[0043] A processor is configured to execute the computer program to implement the steps of the video subtitle generation method described above.

[0044] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the video subtitle generation steps described above.

[0045] The video subtitle generation method provided in this application includes: decomposing a video into a video frame sequence, an audio signal stream, and speech-to-text; extracting features from the video frame sequence, the audio signal stream, and the speech-to-text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively; fusing the visual feature vector sequence, the acoustic feature representation, and the semantic representation to obtain a multimodal video feature representation; performing emotion recognition and quantization based on the multimodal video feature representation to obtain an emotion vector, and obtaining an emotion intensity based on the emotion vector; identifying the speaker's identity and the scene type in the video; selecting subtitle style parameters based on the emotion intensity, the speaker's identity, and the scene type; and rendering subtitles based on the subtitle style parameters.

[0046] As can be seen, the video subtitle generation method provided in this application integrates visual, audio, and textual modal information for analysis, enabling a more comprehensive and accurate understanding of video content. This improves the accuracy and reliability of emotion recognition, person recognition, and scene classification. Furthermore, by selecting subtitle style parameters and rendering subtitles based on emotion intensity, speaker identity, and scene type, the subtitles can accurately match the emotional expression of the video content, enhancing the subtitle effect. Moreover, the entire process requires no manual intervention, improving efficiency and reducing costs.

[0047] The video subtitle generation apparatus, equipment, and computer-readable storage medium provided in this application all have the aforementioned technical effects. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the prior art and embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a video subtitle generation method provided in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of a video subtitle generation device provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] The core of this application is to provide a video subtitle generation method, apparatus, device, and storage medium that can improve efficiency, reduce costs, and enhance subtitle quality.

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a video subtitle generation method provided in an embodiment of this application. (Refer to...) Figure 1 As shown, the method includes:

[0055] S101: Decompose the video into a sequence of video frames, an audio signal stream, and speech-to-text.

[0056] The video subtitle generation method provided in this application is used for generating subtitles for micro-dramas and can be widely applied to online micro-drama production platforms, micro-film creation tools, vertical short video editors, micro-drama content processing by streaming media service providers, online video editing services, and assistive hearing devices. The method decodes and preprocesses the input original micro-drama video, decomposing the preprocessed video into a video frame sequence, an audio signal stream, and speech-to-text.

[0057] S102: Extract features from the video frame sequence, the audio signal stream, and the speech-to-text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively.

[0058] In some embodiments, feature extraction is performed on the video frame sequence, the audio signal stream, and the speech-transcribed text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively, including:

[0059] The spatial feature representation of each frame in the video frame sequence is obtained by an image feature extractor, and the temporal relationship between frames is modeled by a long short-term memory network or a spatiotemporal convolutional network to obtain the visual feature vector sequence.

[0060] The audio signal stream is subjected to frame-by-frame windowing processing, and the Mel spectrogram and Mel frequency cepstral coefficients are extracted as the acoustic feature representation;

[0061] The transcribed text is segmented and vectorized, and a language model is used to obtain the semantic representation.

[0062] For video frame sequences, frames are extracted according to a preset sampling rate, which can be set by comprehensively considering computational efficiency and content coverage. Preferably, a sampling strategy of 2 to 5 frames per second can be used. For audio signal streams, noise reduction and normalization are performed, converting them into PCM or Mel spectrogram formats suitable for subsequent analysis. In addition, video metadata can be analyzed, including basic information such as video resolution, frame rate, duration, and aspect ratio, providing parameter basis for subsequent processing.

[0063] Spatial and temporal features are extracted from video frames. Specifically, a pre-trained image feature extractor can be used to obtain the spatial feature representation of each frame. Then, a long short-term memory network or a spatiotemporal convolutional network is used to model the temporal relationship between frames, and finally, a sequence of visual feature vectors is output.

[0064] The audio signal is processed by frame segmentation and windowing, and the Mel spectrogram and Mel frequency cepstral coefficients are extracted as acoustic feature representations. The temporal patterns of the audio can be learned using a one-dimensional convolutional network.

[0065] The text feature extraction module performs word segmentation and vectorization on the speech-to-text, and uses a pre-trained language model to obtain the semantic representation of the text.

[0066] S103: The visual feature vector sequence, the acoustic feature representation, and the semantic representation are fused to obtain a multimodal video feature representation.

[0067] After extracting features from each modality, a multimodal feature fusion network is used, employing a cross-modal attention mechanism to model the correlations between features from different modalities, generating a unified multimodal video feature representation. This multimodal video feature representation integrates the video's visual content, audio features, and semantic information, providing rich input for subsequent sentiment analysis and scene recognition.

[0068] S104: Perform emotion recognition and quantization based on the multimodal video feature representation to obtain an emotion vector, and obtain the emotion intensity based on the emotion vector.

[0069] Step S104 aims to identify and quantify emotional states based on multimodal video feature representation.

[0070] An emotion vector comprises multiple dimensions. The emotion intensity can be obtained by weighted summation of these dimensions.

[0071] In some embodiments, the emotion vector includes three dimensions: pleasure, activation, and dominance; the pleasure reflects the positive or negative tendency of the emotion; the activation reflects the intensity of the emotion; and the dominance reflects the active or passive characteristics of the emotion.

[0072] An emotion classification model is established to map video content to a predefined emotion category system. This embodiment uses an emotion classification model with three dimensions: pleasure, activation, and dominance. Each dimension is quantified using continuous numerical values. Pleasure reflects the positive or negative tendency of the emotion, with a value range of -1 to +1; high pleasure corresponds to positive emotions, and low pleasure corresponds to negative emotions. Activation reflects the intensity of the emotion, with a value range of zero to one; high activation corresponds to excitement, and low activation corresponds to calmness or depression. Dominance reflects the active or passive characteristics of the emotion, with a value range of zero to one, assisting in judging the speaker's emotional state.

[0073] The emotional intensity is obtained by weighting and summing the pleasure level, activation level, and dominance level. The weights for pleasure level, activation level, and dominance level can be determined in advance through simulation or set based on experience.

[0074] S105: Identify the speaker's identity and the scene type in the video.

[0075] In some embodiments, identifying the speaker in a video includes:

[0076] The speaker's identity in the video is identified using face detection, voice separation, and lip reading technologies.

[0077] This embodiment comprehensively utilizes three techniques—face detection, voice separation, and lip reading—to identify the speaker. For face detection, a deep learning-based face detector is used to locate facial regions in the video, and a face recognition model is used to extract facial feature vectors for identity matching. For voice separation, a speaker separation model is used to decompose mixed audio into independent speech streams from different speakers, and speaker verification is performed using voiceprint features. For lip reading, in cases where clear audio cannot be obtained, the speaker's lip movements are analyzed to assist in identifying the speech content. Ultimately, the speaker detection results, speaker identification, and speaker switching events are obtained.

[0078] In some embodiments, the scene type identified in the video includes:

[0079] Segment the video into semantically coherent scene units;

[0080] Each scene unit is classified by scene type.

[0081] The video is segmented into semantically coherent scene units, and then each scene unit is categorized by type. Scene types can include comedy, tragedy, suspense, romance, action, heartwarming, tense, and transitional scenes. A hierarchical classification strategy can be adopted, starting with coarse-grained classification to determine whether the scene is dialogue-based or non-dialogue-based. For non-dialogue-based scenes, further subdivisions can be made into action, lyrical, and tense categories; for dialogue-based scenes, further subdivisions can be made based on the number of people speaking, the atmosphere, and the content.

[0082] S106: Select subtitle style parameters based on the emotional intensity, the speaker's identity, and the scene type.

[0083] In some embodiments, selecting subtitle style parameters based on the emotional intensity, the speaker's identity, and the scene type includes:

[0084] Select font parameters based on the intensity of the emotion and the type of the scene;

[0085] Select color parameters based on the intensity of the emotion and the speaker's identity;

[0086] Select the location parameters according to the scene type;

[0087] Select the motion effect parameters based on the intensity of the emotion and the type of the scene.

[0088] The subtitle style parameters include font parameters, color parameters, position parameters, and animation parameters.

[0089] Font parameters include three sub-parameters: font family, weight, and size. Choose an appropriate font family based on emotional intensity and scene type. For high-activity scenes, use bold or sans-serif fonts to enhance visual impact; for low-activity scenes, use regular or light fonts to maintain visual softness. For comedic scenes, choose rounded and cute fonts to add fun. For tragic scenes, choose elegant and dignified fonts to enhance emotional expression; for suspenseful scenes, choose concise and sharp fonts to increase tension. Font size can be adjusted according to the importance and activity level of the subtitle content, using larger font sizes for emphasized content.

[0090] The color parameters include three sub-parameters: foreground color, background color, and outline color. The color scheme is selected based on the level of pleasure. Positive emotions use warm color schemes, including hues such as red, orange, and yellow; negative emotions use cool color schemes, including hues such as blue, purple, and green. Activation level affects the saturation and brightness of colors; high-activation scenes use vibrant colors with high saturation and brightness, while low-activation scenes use soft colors with low saturation. Different colors are assigned to different speakers to enhance differentiation; main characters can use relatively eye-catching colors such as gold or red, while supporting characters can use relatively understated colors such as gray or light blue. Additionally, the contrast between the subtitle area and the background can be analyzed in real time, automatically adjusting the subtitle color to ensure clear visibility in various scenarios.

[0091] The position parameters include three sub-parameters: horizontal position, vertical position, and offset. The subtitle position is dynamically adjusted based on the scene type and content. In addition to the traditional bottom-centered position, an intelligent avoidance function is added, automatically detecting important elements in the frame such as faces, key props, and text information, and placing the subtitles in areas that do not obstruct viewing. For emphasized content, subtitles can be placed in the center of the screen or in specific locations to attract attention. For multi-speaker dialogue scenes, a split-column layout strategy is adopted, with subtitles for different speakers appearing near their respective speakers. For vertical short drama scenes, the position parameter settings are optimized to adapt to the 9:16 aspect ratio.

[0092] The animation parameters include three sub-parameters: entrance animation, exit animation, and sustained animation. The animation type and parameter values ​​are configured based on the activation level and scene atmosphere. High-activation scenes use strong entrance animations, such as bouncing, flashing, scaling, and shaking effects. Low-activation scenes use gentler animations, such as fade-in / fade-out effects and smooth movement effects. The duration of the animation is adjusted according to the activation level; shorter animation durations are used in high-activation scenes to enhance the sense of immediacy, while longer animation durations are used in low-activation scenes to maintain visual continuity. The easing curve parameter is selected according to the scene type: comedic scenes use elastic easing curves to increase fun, action scenes use acceleration curves to increase impact, and lyrical scenes use sinusoidal easing curves to increase gentleness.

[0093] This application's embodiments dynamically generate a complete subtitle style scheme encompassing four dimensions: font, color, position, and animation, based on analysis results across three dimensions: emotional intensity, speaker identity, and scene type. This allows subtitles to accurately match the emotional expression of the video content, significantly enhancing the viewing experience.

[0094] S107: Render the subtitles according to the subtitle style parameters.

[0095] In some embodiments, rendering subtitles according to the subtitle style includes:

[0096] Synchronize and align subtitle segments with video frames based on timeline information;

[0097] Generate the corresponding subtitle layer based on the subtitle style parameters of each subtitle;

[0098] Composite the subtitle layer with the original video footage;

[0099] For subtitles that require animation, keyframe animations are generated based on the animation parameters and played according to the preset timeline during video playback.

[0100] Based on the selected subtitle style parameters, the subtitle content is rendered and output in a way that matches the mood of the current scene. First, the subtitle segments are synchronized with the video frames according to the timeline information to ensure that the display time of the subtitles precisely matches the spoken content. Then, according to the style parameter configuration of each subtitle, the rendering engine is invoked to generate the corresponding subtitle layer. The rendering process includes steps such as font rendering, color filling, position transformation, and motion effect binding. For subtitles requiring animation, keyframe animations are generated according to the animation parameters and played according to the set timeline during video playback. After all subtitles are rendered, the subtitle layers are composited with the original video footage. The compositing process needs to handle details such as blending the subtitles with the background, edge anti-aliasing, and transparency blending to ensure that the final output maintains both the clarity and readability of the subtitles and does not affect the viewing experience of the original video.

[0101] The video subtitle generation method provided in the above embodiments of this application can be implemented based on a multimodal video subtitle generation and emotional typesetting system. The main steps of video subtitle generation and emotional typesetting include video input and preprocessing, multimodal feature extraction and fusion, emotion intensity analysis and quantification, speaker identification and speaker annotation, scene type classification and context modeling, dynamic subtitle style parameter generation, emotional typesetting rendering and subtitle overlay.

[0102] Multimodal feature extraction and fusion can be achieved based on a multimodal feature extraction network and a multimodal feature fusion network. Multimodal feature extraction comprises three parallel branches: a visual feature extraction module, an audio feature extraction module, and a text feature extraction module. The visual feature extraction module uses a convolutional neural network as its backbone to extract spatial and temporal features from video frames. Specifically, it uses a pre-trained image feature extractor to obtain the spatial feature representation of each frame, and then models the temporal relationships between frames using a long short-term memory network or a spatiotemporal convolutional network, ultimately outputting a sequence of visual feature vectors. The audio feature extraction module performs frame-by-frame windowing processing on the audio signal, extracting Mel spectrograms and Mel frequency cepstral coefficients as acoustic feature representations, and uses a one-dimensional convolutional network to learn the temporal patterns of the audio. The text feature extraction module performs word segmentation and vectorization processing on the speech-to-text, and uses a pre-trained language model to obtain the semantic representation of the text. After completing the extraction of features from each modality, a multimodal feature fusion network is used, employing a cross-modal attention mechanism to model the correlations between features from different modalities, generating a unified multimodal video feature representation.

[0103] Emotion intensity analysis and quantification can be achieved by the emotion intensity analysis module. This module identifies and quantifies emotional states based on multimodal video feature representations. First, it establishes an emotion classification model, mapping video content to a predefined emotion category system. The emotion classification model includes three dimensions: pleasure, activation, and dominance. Each dimension is quantified using continuous numerical values. Pleasure reflects the positive or negative tendency of the emotion, ranging from -1 to +1; high pleasure corresponds to positive emotions, and low pleasure corresponds to negative emotions. Activation reflects the intensity of the emotion, ranging from zero to one; high activation corresponds to excitement, and low activation corresponds to calmness or depression. Dominance reflects the active or passive characteristics of the emotion, ranging from zero to one, assisting in judging the speaker's emotional state. The input to the emotion intensity analysis module is the multimodal fusion feature representation, and the output is a three-dimensional emotion vector at the current time point. The emotion classification model employs a multi-task learning network architecture, simultaneously performing emotion classification and emotion dimension regression. The classification task determines the main emotion categories, while the regression task accurately quantifies the intensity values ​​of each dimension.

[0104] Speaker identification and annotation can be implemented by the speaker identification module. This module detects and identifies speakers in the video. It integrates three technologies: face detection, voice separation, and lip reading. For face detection, a deep learning-based face detector locates facial regions in the video, and a face recognition model extracts feature vectors for identity matching. For voice separation, a speaker separation model decomposes mixed audio into independent speech streams from different speakers, and combines voiceprint features for speaker verification. For lip reading, in cases where clear audio is unavailable, analysis of the speaker's lip movements assists in identifying the speech content. The output of the speaker identification module includes speaker detection results, speaker identification identifiers, and speaker switching events. When a speaker switch is detected, the module generates a corresponding switch event marker, providing a basis for subsequent subtitle style differentiation processing.

[0105] Scene type classification and context modeling can be implemented by the scene type classification module. This module automatically identifies the scene type of the video. First, it segments the video into semantically coherent scene units, and then classifies each scene unit by type. Considering the unique characteristics of micro-drama content, this embodiment establishes a scene type system specifically for micro-drama scenes, covering the following main categories: comedy scenes, tragedy scenes, suspense scenes, romance scenes, action scenes, heartwarming scenes, tense scenes, and transitional scenes. The scene type classification module adopts a hierarchical classification strategy. First, it performs coarse-grained classification to determine whether the scene belongs to dialogue or non-dialogue categories. For non-dialogue scenes, it further subdivides them into action, lyrical, and tense categories. For dialogue scenes, it further subdivides them based on the number of people speaking, the atmosphere, and the content.

[0106] The generation of dynamic subtitle style parameters can be achieved by the dynamic style parameter generation module.

[0107] The dynamic style parameter generation module generates personalized style parameter configurations for each subtitle segment. This module establishes a mapping rule base from analysis results to style parameters, including the following parameter dimensions:

[0108] Font parameters include three sub-parameters: font family, font weight, and font size. Choose an appropriate font family based on emotional intensity and scene type: use bold or sans-serif fonts for high-activity scenes to enhance visual impact, and use regular or light fonts for low-activity scenes to maintain visual softness; for comedic scenes, choose rounded and cute fonts to add fun; for tragic scenes, choose elegant and dignified fonts to enhance emotional expression; for suspenseful scenes, choose concise and sharp fonts to increase tension. Adjust the font size according to the importance and emotional intensity of the subtitle content, using larger font sizes for emphasized content.

[0109] The color parameters include three sub-parameters: foreground color, background color, and outline color. The color scheme is selected based on the level of pleasantness or unpleasantness of the emotion: positive emotions use warm color schemes, including hues such as red, orange, and yellow; negative emotions use cool color schemes, including hues such as blue, purple, and green. Activation level affects color saturation and brightness; high-activation scenes use vibrant colors with high saturation and brightness, while low-activation scenes use soft colors with low saturation. Different colors are assigned to different speakers to enhance differentiation; main characters use relatively eye-catching colors such as gold or red, while supporting characters use relatively understated colors such as gray or light blue. Simultaneously, the color parameter generation module analyzes the contrast between the subtitle area and the background in real time, automatically adjusting the subtitle color to ensure clear visibility in various scenarios.

[0110] The position parameters include three sub-parameters: horizontal position, vertical position, and offset. The subtitle position is dynamically adjusted based on the scene type and content. In addition to the traditional bottom-centered position, an intelligent avoidance function is added, automatically detecting important elements in the frame such as faces, key props, and text information, and placing the subtitles in areas that do not obstruct viewing. For emphasized content, subtitles can be placed in the center of the screen or in specific locations to attract attention. For multi-speaker dialogue scenes, a split-column layout strategy is adopted, with subtitles for different speakers appearing near their respective speakers. For vertical short drama scenes, the position parameter settings are optimized to adapt to the 9:16 aspect ratio.

[0111] The animation parameters include three sub-parameters: entrance animation, exit animation, and sustained animation. The animation type and parameter values ​​are configured based on the emotional intensity and scene atmosphere. High-emotional-intensity scenes use strong entrance animations, such as bouncing, flashing, scaling, and shaking effects; low-emotional-intensity scenes use gentler animations, such as fade-in / fade-out effects and smooth movement effects. The duration of the animation is adjusted according to the emotional intensity; short animation durations are used for high-emotional-intensity scenes to enhance the sense of immediacy, while longer animation durations are used for low-emotional-intensity scenes to maintain visual continuity. The easing curve parameter is selected according to the scene type: comedic scenes use elastic easing curves to increase humor, action scenes use acceleration curves to increase impact, and lyrical scenes use sinusoidal easing curves to increase gentleness.

[0112] Emotional typesetting rendering and subtitle overlay can be achieved by an emotional typesetting engine.

[0113] The emotional typesetting engine renders subtitles in a way that matches the mood of the current scene. First, it synchronizes subtitle segments with video frames based on timeline information, ensuring the subtitle display time precisely matches the spoken content. Then, based on the style parameters configured for each subtitle, it calls the rendering engine to generate the corresponding subtitle layer. The rendering process includes font rendering, color filling, position transformation, and motion effect binding. For subtitles requiring animation, the emotional typesetting engine generates keyframe animations based on motion effect parameters and plays them according to the set timeline during video playback. After rendering all subtitles, the emotional typesetting engine composites the subtitle layers with the original video footage. The compositing process handles details such as blending subtitles with the background, edge anti-aliasing, and transparency blending to ensure the final output maintains both clear and readable subtitles without affecting the viewing experience of the original video.

[0114] For ease of understanding, the following describes several examples of subtitle generation for different types of short dramas.

[0115] Subtitle generation for urban romance short dramas:

[0116] This example uses a short video with an urban romance theme. The video is approximately 3 minutes long and includes the following main scenes: an opening encounter, a misunderstanding and conflict, a confession of feelings, and a heartwarming ending. The video involves two main characters: the female protagonist and the male protagonist.

[0117] Processing Procedure: In the opening encounter scene, the visual feature extraction module detected visual features of the café environment, including soft lighting, wooden furniture, and warm-toned decor; the audio feature extraction module identified gentle background music and slow ambient sounds; and the text feature extraction module obtained the content of the characters' initial dialogue. The sentiment analysis module output a three-dimensional sentiment vector with a pleasure level of 0.6, an activation level of 0.3, and a dominance level of 0.4, indicating that the scene has a moderately pleasant, low-activation, and warm atmosphere.

[0118] In the misunderstanding and conflict scenario, the visual feature extraction module detected tense changes in the characters' facial expressions and increased amplitude of their movements; the audio feature extraction module identified the accelerated rhythm of the background music and emotional tone changes; and the text feature extraction module obtained dialogue content with emotional overtones. The emotion analysis module outputs a three-dimensional emotion vector with a pleasure level of -0.3, an activation level of 0.8, and a dominance level of 0.7, indicating that the scenario is a tense atmosphere of low pleasure and high activation.

[0119] In the emotional confession scene, the visual feature extraction module detected the romanticization of the composition and the softening of the lighting; the audio feature extraction module identified the climax of the background music and the emotional tone features; and the text feature extraction module obtained the core lines of the confession. The emotion analysis module output a three-dimensional emotion vector with a pleasure level of 0.9, an activation level of 0.7, and a dominance level of 0.5, indicating that the scene is a strong emotional atmosphere of high pleasure and high activation.

[0120] In the heartwarming ending scene, the visual feature extraction module detected the warming of the screen's color tone and the relaxation of the characters' expressions; the audio feature extraction module identified the soothing nature of the background music; and the text feature extraction module obtained the dialogue content indicating reconciliation. The emotion analysis module output a three-dimensional emotion vector with a pleasure level of 0.8, an activation level of 0.2, and a dominance level of 0.3, indicating that the scene has a warm atmosphere of high pleasure and low activation.

[0121] Subtitle style parameter generation:

[0122] Based on the sentiment analysis results above, the dynamic style parameter generation module generates style configurations for each scenario:

[0123] In the opening encounter scene, the font uses a rounded font with a regular weight and a medium size; the color is a warm orange foreground color with an opacity of 0.9 to maintain a soft feel; the position is centered at the bottom; and the animation uses a slow fade-in effect with a duration of 500 milliseconds.

[0124] In misunderstanding and conflict scenarios, the font is switched to bold and the font size is increased by 20%; the color is switched to dark red foreground to enhance the tension; the position remains centered at the bottom, but the upward offset is increased to avoid the possible facial area of ​​the characters; the animation uses a rapid bouncing effect with a duration of 200 milliseconds.

[0125] In emotional confession scenes, the font is switched to an elegant serif font, and the font size is increased by 10%; a magenta foreground color is used to enhance the romantic feel; the position is centered to highlight key lines; the animation uses a word-by-word typewriter effect, with a duration of 80 milliseconds per word, and a soft breathing and flickering effect is added after all words are displayed.

[0126] In the heartwarming ending scene, the font is switched to a thin font, and the font size is restored to medium; the color is a soft, warm yellow foreground color; the position is restored to the bottom center; the animation uses a slow fade-in / fade-out effect with a duration of 600 milliseconds.

[0127] Processing effect:

[0128] The subtitles in this urban romance short drama present a visual effect that highly matches the emotions of the content. Viewers can clearly feel the harmony between the style of the subtitles and the mood of the visuals. The typewriter effect in the confession scene enhances the dramatic tension, the deep red subtitles in the misunderstanding and conflict scenes deepen the tense atmosphere, and the soft animation of the heartwarming ending creates a comfortable viewing experience.

[0129] Subtitle generation for suspenseful mystery short dramas:

[0130] This example uses a short suspense / mystery video as an illustration. The video is approximately 4 minutes long and includes the following main scenes: the discovery of the case, the reasoning and analysis, the presentation of key evidence, and the revelation of the truth. The short drama involves three main characters: Detective Li Ming, suspect Wang Fang, and witness Zhang Qiang.

[0131] Processing procedure:

[0132] In the crime scene, the visual feature extraction module detects detailed features of the scene, including the placement of objects and the distribution of traces; the audio feature extraction module identifies low-pitched ambient sound effects and the detective's observational narration; and the text feature extraction module obtains objective, descriptive dialogue. The sentiment analysis module outputs a three-dimensional sentiment vector with a pleasure level of -0.2, an activation level of 0.5, and a dominance level of 0.6.

[0133] In the reasoning and analysis scenario, the visual feature extraction module detects the presence of analytical materials such as mind maps and photographic evidence; the audio feature extraction module identifies background music with a strong rhythm; and the text feature extraction module obtains logically reasoning dialogue content. The sentiment analysis module outputs a three-dimensional sentiment vector with a pleasure level of 0.0, an activation level of 0.6, and a dominance level of 0.7.

[0134] In the critical evidence scenario, the visual feature extraction module detected highlighted evidence and close-up shots; the audio feature extraction module identified abrupt music stops and changes in sound effects; and the text feature extraction module obtained key dialogue content. The sentiment analysis module output a three-dimensional sentiment vector with a pleasure level of -0.4, an activation level of 0.9, and a dominance level of 0.8, indicating that the scenario was a highly tense and critical moment.

[0135] In the truth-revealing scenario, the visual feature extraction module detects the visual changes and characters' reactions as the truth is revealed; the audio feature extraction module identifies the climax of the music; and the text feature extraction module obtains the core dialogue that reveals the truth. The emotion analysis module outputs a three-dimensional emotion vector with a pleasure level of 0.7, an activation level of 0.8, and a dominance level of 0.9.

[0136] Speaker identification and differentiation:

[0137] The speaker identification module identifies and tracks three characters in the video: dark blue subtitles are assigned to Detective Li Ming; purple subtitles to Suspect Wang Fang; and gray subtitles to Witness Zhang Qiang. During dialogue, when characters switch, the subtitle color automatically changes to the corresponding character's color, requiring no manual intervention.

[0138] Subtitle style parameter generation:

[0139] Based on the sentiment analysis results and speaker identity information, the dynamic style parameter generation module generates style configurations for each scenario:

[0140] In the crime scene, a simple sans-serif font is used with a standard size; a cool gray foreground color is used to maintain objectivity; the font is centered at the bottom; and a simple fade-in effect is used for the animation.

[0141] In reasoning and analysis scenarios, a monospaced font is used to enhance the sense of logic, while maintaining a standard font size; a blue color scheme is used to enhance the sense of technology; the position is shifted to the left to leave space for possible evidence annotations; and a fast-switching animation effect is used to match the rhythm of reasoning.

[0142] In the key evidence scene, the font is switched to bold black, and the font size is increased by 30% to enhance the visual impact; the color is switched to a high-contrast white foreground with a black outline; the position is adjusted to the center of the screen to attract attention; the animation uses a strong scaling and bouncing effect with a duration of 150 milliseconds.

[0143] In the scene where the truth is revealed, a dignified serif font is used, with the font size increased by 20%; a gold foreground color is used to enhance the dramatic effect; the font is centered; and the animation effect is a typewriter effect that displays each character sequentially to complement the final revelation.

[0144] Processing effect:

[0145] The subtitles in this suspenseful mystery short drama present a professional and concise visual effect. The automatic character differentiation allows viewers to quickly identify the speaker's identity, the strong motion effects in key evidence scenes enhance the dramatic tension, and the concise style of the reasoning and analysis scenes maintains the professionalism of the content.

[0146] Comedy and humorous short drama subtitle generation:

[0147] This example uses a short comedy video as an illustration. The video is about 2 minutes long and features one person playing multiple roles, including exaggerated expressions, unexpected plot twists, and nonsensical storylines.

[0148] Processing procedure:

[0149] The generation of subtitles for comedy micro-dramas requires special consideration of humor and exaggeration. The sentiment analysis module not only analyzes basic emotional dimensions but also introduces a humor index to assess the content's comedic level.

[0150] In typical conversational scenarios, the sentiment analysis module outputs a three-dimensional vector with a pleasure level of 0.5, an activation level of 0.4, and a humor level of 0.3.

[0151] In comedic plot twists, the sentiment analysis module outputs a three-dimensional vector with a pleasure level of 0.7, an activation level of 0.7, and a humor level of 0.8, indicating that the scene is highly comedic.

[0152] In exaggerated performance scenarios, the emotion analysis module outputs a three-dimensional vector with a pleasure level of 0.8, an activation level of 0.9, and a humor level of 0.9, indicating that the scenario is extremely exaggerated comedic content.

[0153] Subtitle style parameter generation:

[0154] Based on the humor index, the dynamic style parameter generation module generates more interesting style configurations for each scenario:

[0155] In normal dialogue scenarios, a rounded and cute font is used, with the font size remaining standard; a bright orange foreground color is used; and simple fade-in and fade-out effects are used for animation.

[0156] In comedic plot twists, the font uses a handwritten style to increase friendliness; the color scheme uses vibrant yellow or green foreground colors; and the animation uses bouncing or shaking effects to match the comedic atmosphere.

[0157] In exaggerated performance scenes, the font uses the boldest round font, with the font size increased by 25%; the color uses a highly saturated rainbow gradient effect; the position can appear anywhere on the screen to match the exaggerated performance; the motion effects use exaggerated bouncing and rotating effects, combined with flashing or star effects.

[0158] Processing effect:

[0159] Through the processing of this invention, the subtitles of this comedic micro-drama present a lively and exaggerated visual effect, highly matching the comedic style of the content. The rainbow gradient subtitle colors and exaggerated bouncing animation enhance the entertainment value of the content and increase the audience's ability to laugh.

[0160] In summary, the video subtitle generation method provided in this application integrates visual, audio, and textual modal information for analysis, enabling a more comprehensive and accurate understanding of video content. This improves the accuracy and reliability of emotion recognition, character recognition, and scene classification. Furthermore, by selecting subtitle style parameters and rendering subtitles based on emotion intensity, speaker identity, and scene type, the subtitles can accurately match the emotional expression of the video content, enhancing the subtitle effect. Moreover, the entire process requires no manual intervention, improving efficiency and reducing costs. Applying this method can achieve a balance between real-time performance and quality while ensuring the processing effect of high-emotion-density micro-drama content, significantly improving the viewing experience and information delivery effect of micro-drama subtitles.

[0161] This application also provides a video subtitle generation apparatus, which is described below and can be referred to in conjunction with the method described above. Please refer to... Figure 2 , Figure 2 This is a schematic diagram of a video subtitle generation device provided in an embodiment of this application, combined with... Figure 2 As shown, the device includes:

[0162] Decomposition unit 10 is used to decompose the video into a video frame sequence, an audio signal stream, and speech-to-text.

[0163] Extraction unit 20 is used to extract features from the video frame sequence, the audio signal stream, and the speech-to-text, and obtain visual feature vector sequence, acoustic feature representation, and semantic representation, respectively;

[0164] The fusion unit 30 is used to fuse the visual feature vector sequence, the acoustic feature representation, and the semantic representation to obtain a multimodal video feature representation;

[0165] The emotion recognition unit 40 is used to perform emotion recognition and quantization based on the multimodal video feature representation to obtain an emotion vector, and to obtain the emotion intensity based on the emotion vector;

[0166] Scene recognition unit 50 is used to identify the speaker's identity and the scene type of the video;

[0167] Selection unit 60 is used to select subtitle style parameters based on the emotion intensity, the speaker's identity, and the scene type;

[0168] The rendering unit 70 is used to render subtitles according to the subtitle style parameters.

[0169] In some embodiments, the emotion vector includes three dimensions: pleasure, activation, and dominance; the pleasure reflects the positive or negative tendency of the emotion; the activation reflects the intensity of the emotion; and the dominance reflects the active or passive characteristics of the emotion.

[0170] In some embodiments, the scene recognition unit 50 is used for:

[0171] The speaker's identity in the video is identified using face detection, voice separation, and lip reading technologies.

[0172] In some embodiments, the scene recognition unit 50 is used for:

[0173] Segment the video into semantically coherent scene units;

[0174] Each scene unit is classified by scene type.

[0175] In some embodiments, the selection unit 60 is used for:

[0176] Select font parameters based on the intensity of the emotion and the type of the scene;

[0177] Select color parameters based on the intensity of the emotion and the speaker's identity;

[0178] Select the location parameters according to the scene type;

[0179] Select the motion effect parameters based on the intensity of the emotion and the type of the scene.

[0180] In some embodiments, the rendering unit 70 is used for:

[0181] Synchronize and align subtitle segments with video frames based on timeline information;

[0182] Generate the corresponding subtitle layer based on the subtitle style parameters of each subtitle;

[0183] Composite the subtitle layer with the original video footage;

[0184] For subtitles that require animation, keyframe animations are generated based on the animation parameters and played according to the preset timeline during video playback.

[0185] In some embodiments, the extraction unit 20 is used for:

[0186] The spatial feature representation of each frame in the video frame sequence is obtained by an image feature extractor, and the temporal relationship between frames is modeled by a long short-term memory network or a spatiotemporal convolutional network to obtain the visual feature vector sequence.

[0187] The audio signal stream is subjected to frame-by-frame windowing processing, and the Mel spectrogram and Mel frequency cepstral coefficients are extracted as the acoustic feature representation;

[0188] The transcribed text is segmented and vectorized, and a language model is used to obtain the semantic representation.

[0189] This application also provides an electronic device, referenced... Figure 3 As shown, the device includes a memory 1 and a processor 2.

[0190] Memory 1 is used to store computer programs;

[0191] Processor 2 is used to execute computer programs to perform the following steps:

[0192] The video is decomposed into a video frame sequence, an audio signal stream, and speech-to-text. Feature extraction is performed on the video frame sequence, the audio signal stream, and the speech-to-text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively. The visual feature vector sequence, the acoustic feature representation, and the semantic representation are fused to obtain a multimodal video feature representation. Emotion recognition and quantization are performed based on the multimodal video feature representation to obtain an emotion vector, and the emotion intensity is obtained based on the emotion vector. The speaker's identity and the scene type in the video are identified. Subtitle style parameters are selected based on the emotion intensity, the speaker's identity, and the scene type. Subtitles are rendered based on the subtitle style parameters.

[0193] For a description of the equipment provided in this application, please refer to the above method embodiments; further details will not be provided here.

[0194] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps:

[0195] The video is decomposed into a video frame sequence, an audio signal stream, and speech-to-text. Feature extraction is performed on the video frame sequence, the audio signal stream, and the speech-to-text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively. The visual feature vector sequence, the acoustic feature representation, and the semantic representation are fused to obtain a multimodal video feature representation. Emotion recognition and quantization are performed based on the multimodal video feature representation to obtain an emotion vector, and the emotion intensity is obtained based on the emotion vector. The speaker's identity and the scene type in the video are identified. Subtitle style parameters are selected based on the emotion intensity, the speaker's identity, and the scene type. Subtitles are rendered based on the subtitle style parameters.

[0196] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0197] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0198] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant details can be found in the method section.

[0199] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0201] The video subtitle generation method, apparatus, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for generating video subtitles, characterized in that, include: The video is broken down into a sequence of video frames, an audio signal stream, and speech-to-text transcription. Feature extraction is performed on the video frame sequence, the audio signal stream, and the speech-transcribed text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively. The visual feature vector sequence, the acoustic feature representation, and the semantic representation are fused to obtain a multimodal video feature representation; Emotion vectors are obtained by performing emotion recognition and quantization based on the multimodal video feature representation, and the emotion intensity is obtained based on the emotion vectors. Identify the speaker's identity and the video scene type in the video; Select subtitle style parameters based on the emotional intensity, the speaker's identity, and the scene type; Render the subtitles according to the subtitle style parameters.

2. The video subtitle generation method according to claim 1, characterized in that, The emotion vector includes three dimensions: pleasure, activation, and dominance. Pleasure reflects the positive or negative tendency of the emotion; activation reflects the intensity of the emotion; and dominance reflects the active or passive characteristics of the emotion.

3. The video subtitle generation method according to claim 1, characterized in that, Identifying the speaker in a video includes: The speaker's identity in the video is identified using face detection, voice separation, and lip reading technologies.

4. The video subtitle generation method according to claim 1, characterized in that, The types of scenes that can be identified in videos include: Segment the video into semantically coherent scene units; Each scene unit is classified by scene type.

5. The video subtitle generation method according to claim 2, characterized in that, The subtitle style parameters selected based on the emotional intensity, the speaker's identity, and the scene type include: Select font parameters based on the intensity of the emotion and the type of the scene; Select color parameters based on the intensity of the emotion and the speaker's identity; Select the location parameters according to the scene type; Select the motion effect parameters based on the intensity of the emotion and the type of the scene.

6. The video subtitle generation method according to claim 5, characterized in that, Rendering subtitles according to the aforementioned subtitle style includes: Synchronize and align subtitle segments with video frames based on timeline information; Generate the corresponding subtitle layer based on the subtitle style parameters of each subtitle; Composite the subtitle layer with the original video footage; For subtitles that require animation, keyframe animations are generated based on the animation parameters and played according to the preset timeline during video playback.

7. The video subtitle generation method according to claim 6, characterized in that, Feature extraction is performed on the video frame sequence, the audio signal stream, and the speech-to-text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively: The spatial feature representation of each frame in the video frame sequence is obtained by an image feature extractor, and the temporal relationship between frames is modeled by a long short-term memory network or a spatiotemporal convolutional network to obtain the visual feature vector sequence. The audio signal stream is subjected to frame-by-frame windowing processing, and the Mel spectrogram and Mel frequency cepstral coefficients are extracted as the acoustic feature representation; The transcribed text is segmented and vectorized, and a language model is used to obtain the semantic representation.

8. A video subtitle generation device, characterized in that, include: The decomposition unit is used to decompose video into a sequence of video frames, an audio signal stream, and speech-to-text. The extraction unit is used to extract features from the video frame sequence, the audio signal stream, and the speech-transcribed text to obtain a visual feature vector sequence, an acoustic feature representation, and a semantic representation, respectively. A fusion unit is used to fuse the visual feature vector sequence, the acoustic feature representation, and the semantic representation to obtain a multimodal video feature representation; An emotion recognition unit is used to perform emotion recognition and quantization based on the multimodal video feature representation to obtain an emotion vector, and to obtain the emotion intensity based on the emotion vector; The scene recognition unit is used to identify the speaker's identity and the scene type in the video; The selection unit is used to select subtitle style parameters based on the emotional intensity, the speaker's identity, and the scene type. The rendering unit is used to render subtitles according to the subtitle style parameters.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the video subtitle generation method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the video subtitle generation steps as described in any one of claims 1 to 7.

Images