A video playing method, an electronic device, a storage medium and a computer program product

Abstract

The application provides a video playing method, an electronic device, a storage medium and a computer program product. The method comprises the following steps: acquiring a media stream; the media stream is composed of an audio stream and a video stream; in the case that a speaker's emotional sound feature and / or atmosphere sound feature is detected from the audio stream, determining a text visual parameter and / or a visual element parameter according to the emotional sound feature and / or the atmosphere sound feature; adjusting a visual style of a subtitle of the video stream by using the text visual parameter and / or generating a visual object by using the visual element parameter, and synthesizing the visual style adjusted subtitle and / or the visual object into the video stream to obtain a video stream with the visual style adjusted subtitle and / or the visual object; and outputting the video stream with the visual style adjusted subtitle and / or the visual object to play the video stream with the visual style adjusted subtitle and / or the visual object on a playing interface.

Description

Technical Field

[0001] This application relates to electronic application technology, and more particularly to a video playback method, electronic device, storage medium, and computer program product. Background Technology

[0002] In the current media technology field, subtitle generation is a mature and widely used function. Its core purpose is to convert the audio content in audio and video streams into text to assist viewers in understanding. This technology provides a basic means of information access for the hearing impaired and also meets the needs of ordinary users to watch videos in quiet or noisy environments.

[0003] Currently, subtitles can be given emotional color by performing text sentiment analysis based on Natural Language Processing (NLP), but this results in low accuracy of sentiment recognition. Summary of the Invention

[0004] This application provides a video playback method, an electronic device, a storage medium, and a computer program product.

[0005] The technical solution of this application is implemented as follows: Firstly, this application proposes a video playback method, the method comprising: Acquire media streams; the media streams consist of audio streams and video streams; If the speaker’s emotional voice features and / or ambient voice features are detected from the audio stream, text visual parameters and / or visual element parameters are determined based on the emotional voice features and / or the ambient voice features. The visual style of the subtitles in the video stream is adjusted using the text visual parameters and / or a visual object is generated using the visual element parameters. The visually styled subtitles and / or the visual object are then combined into the video stream to obtain a video stream with visually styled subtitles and / or a visual object. Output the video stream with visually styled subtitles and / or visual objects to play the video stream with visually styled subtitles and / or visual objects on the playback interface.

[0006] Optionally, after acquiring the media stream and before determining the text visual parameters and / or visual element parameters based on the emotional sound features and / or the ambient sound features, the method further includes: The speaker's speech signal is separated from the audio stream; acoustic features for characterizing emotions are extracted from the speech signal, and the acoustic features are analyzed using a deep learning classifier to obtain the speaker's emotion type and emotion intensity; the speaker's emotional voice features are constructed based on the speaker's emotion type and emotion intensity. And / or, in the case of separating ambient sound effects from the audio stream, determining the sound type corresponding to the ambient sound effect and extracting physical parameters related to the sound type from the ambient sound effect, and constructing the ambient sound features based on the sound type and the physical parameters.

[0007] Optionally, the acoustic features consist of prosodic features and / or spectral features, and the deep learning classifier is composed of a convolutional neural network and a long short-term memory network. The analysis of the acoustic features using the deep learning classifier to obtain the speaker's emotion type and emotion intensity includes: The convolutional neural network is used to collect emotion-related patterns from the prosodic features and / or the spectral features; By utilizing the Long Short-Term Memory Network to learn the time-series dependencies of the emotion-related patterns, the speaker's emotion type and intensity can be identified.

[0008] Optionally, determining the sound type corresponding to the ambient sound effect and extracting physical parameters related to the sound type from the ambient sound effect includes: The sound type is determined to be ambient sound. The time spectrum of the ambient sound effect is analyzed by short-time Fourier transform to obtain physical parameters that characterize the intensity of the ambient sound. And / or, determine that the sound type is an effect sound, use autocorrelation function analysis or beat detection algorithm to extract the rhythm data of the environmental sound effect and the instantaneous energy peak of each beat, and determine the rhythm data and the instantaneous energy peak of each beat as physical parameters related to the effect sound.

[0009] Optionally, determining the text visual parameters and / or visual element parameters based on the emotional sound features and / or the ambient sound features includes: From the sound feature and visual parameter mapping rule base, search for the text visual parameters and / or visual element parameters corresponding to the emotional sound features and / or the ambient sound features.

[0010] Optionally, the text visual parameters include, but are not limited to, at least one of the following: font weight, font color, font animation effects, and font special effects; The visual element parameters include, but are not limited to, at least one of the following: shape, speed, color, position, trajectory, amplitude, and animation effect.

[0011] Optionally, before adjusting the visual style of the subtitles in the video stream using the text visual parameters, the method further includes: Identify text information from the audio stream; The text information is used to generate subtitles for the video stream.

[0012] Secondly, this application provides an electronic device comprising: a processor, a memory, and a communication bus; wherein the processor executes a running program stored in the memory to implement the method as described in any one of claims 1-7.

[0013] Thirdly, this application proposes a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described video playback method.

[0014] Fourthly, this application proposes a computer program product, including a computer program that, when executed by a processor, implements the above-described video playback method.

[0015] The embodiments of this application have the following beneficial effects: by detecting the speaker's emotional voice features and / or ambient sound features in the audio stream, text visual parameters and / or visual element parameters are determined to adjust the visual style of the subtitles in the video stream and / or generate visual objects, which are then synthesized into the video stream. This ensures that the subtitles and / or visual objects in the video stream used for playback on the playback interface accurately match the speaker's true emotional state and the atmosphere of the scene, significantly improving the fidelity and dimensionality of information transmission, thereby significantly improving the accuracy of emotion recognition. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a video playback method provided in this application embodiment; Figure 2 A schematic diagram of the structure of an exemplary system for generating emotional subtitles based on multimodal acoustic features, provided for embodiments of this application; Figure 3 An interactive diagram illustrating an exemplary method for generating emotional subtitles based on multimodal acoustic features, provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device provided in this application embodiment. Figure 1 ; Figure 5 A schematic diagram of the structure of an electronic device provided in this application embodiment. Figure 2 .

[0017] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0018] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first, second, third" used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0021] This application provides a video playback method, such as... Figure 1 As shown, the method may include: S101. Obtain the media stream; the media stream consists of an audio stream and a video stream.

[0022] The video playback method proposed in this application can be applied to scenarios such as live streaming with subtitles or other video playback.

[0023] In this embodiment of the application, after the system is started, it receives a real-time or pre-recorded media stream containing multi-channel audio streams and video streams from the media source.

[0024] S102. If the speaker's emotional voice features and / or ambient voice features are detected from the audio stream, determine the text visual parameters and / or visual element parameters based on the emotional voice features and / or ambient voice features.

[0025] In this embodiment, the audio stream in the media stream is processed by performing two parallel tasks. First, blind source separation (BSS) or a deep learning-based sound source separation algorithm is used to decompose the mixed audio stream into independent audio tracks, such as the speaker's voice signal and ambient sound effects. The ambient sound effects can include ambient sounds and other sound effects. Second, automatic speech recognition (ASR) is performed on the separated speaker's voice signal track to identify the corresponding text information from the audio stream.

[0026] In this embodiment of the application, after separating the speaker's voice signal and ambient sound effects from the audio stream, the speaker's emotional voice features are detected from the speaker's voice signal, and ambient sound features are extracted from the ambient sound effects.

[0027] In one embodiment, the speaker's speech signal is separated from the audio stream; when acoustic features for characterizing emotions are extracted from the speech signal, a deep learning classifier is used to analyze the acoustic features to obtain the speaker's emotion type and emotion intensity; and the speaker's emotional voice features are constructed based on the speaker's emotion type and emotion intensity.

[0028] It should be noted that the embodiments of this application are used for in-depth analysis of the speaker's emotional state.

[0029] In this embodiment, the speaker's speech signal is first preprocessed, including operations such as framing and windowing, and then an acoustic feature that can characterize emotion is extracted from the preprocessed speech signal.

[0030] In the embodiments of this application, acoustic features may consist of prosodic features and / or spectral features. Prosodic features may include, but are not limited to, at least one of the following: the mean, range, and profile of the fundamental frequency (Pitch), dynamic changes in energy, and speech rate; spectral features may include, but are not limited to, at least one of the following: Mel-Frequency Cepstral Coefficients (MFCCs) and their first and second-order differences, etc. These features can effectively describe changes in timbre.

[0031] In this embodiment of the application, a deep learning classifier is pre-trained, wherein the deep learning classifier may be composed of a convolutional neural network and a long short-term memory network.

[0032] In this embodiment, a convolutional neural network is used to collect emotion-related patterns from prosodic features and / or spectral features; and a long short-term memory network is used to learn the time-series dependencies of emotion-related patterns in order to identify the speaker's emotion type and emotion intensity.

[0033] In the embodiments of this application, the emotion type may include complex emotions such as excitement, sadness, tension, and indifference.

[0034] In this embodiment of the application, the speaker's emotional voice feature E_vector can be represented as E_vector=(C_emotion,I_intensity), where C_emotion is the speaker's emotion type, C_emotion is a discrete emotion category (e.g., Excitement, Sadness, Tension), and I_intensity is the emotion intensity, representing the continuous value of the emotion intensity (e.g., normalized to the [0,1] interval).

[0035] In another embodiment, when the ambient sound effect is separated from the audio stream, the sound type corresponding to the ambient sound effect is determined and physical parameters related to the sound type are extracted from the ambient sound effect, and ambient sound features are constructed based on the sound type and physical parameters.

[0036] It should be noted that the embodiments of this application are used to quantify the physical properties and spatial sense of sound in order to construct a visual expression of atmosphere.

[0037] In this embodiment of the application, the ambient sound effect may include ambient sound, sound effect sound, etc., wherein the ambient sound effect may be a continuous ambient sound such as cheers and applause, and the sound effect sound may include a strong rhythmic sound effect such as heartbeat and drumbeat.

[0038] It should be noted that different methods are used to detect ambient sound features for different types of environmental sound effects in this application embodiment.

[0039] In this embodiment of the application, the sound type is determined to be ambient sound, and the time spectrum of the ambient sound effect is analyzed by short-time Fourier transform to obtain physical parameters used to characterize the intensity of the ambient sound; and / or, the sound type is determined to be effect sound, and the rhythm data (such as beat rate) and the instantaneous energy peak of each beat of the ambient sound effect are extracted by autocorrelation function analysis or beat detection algorithm, and the rhythm data and the instantaneous energy peak of each beat are determined as physical parameters related to the effect sound.

[0040] In the embodiments of this application, the physical parameters used to characterize the intensity of ambient sound may include, but are not limited to, at least one of the following: overall loudness (measured in full-scale units, LUFS) as a function of time, the spectral centroid, and the distribution of energy in different frequency bands (frequency distribution). These parameters together constitute a quantitative description of the intensity and atmosphere of ambient sound, such as a quantitative description of the intensity and "enthusiasm" of "cheers".

[0041] In this embodiment, the ambient sound feature S_vector can be represented as: S_vector=(T_sound, A_params). Wherein, T_sound is the sound type (e.g., Cheer, Heartbeat), and A_params is a set containing all relevant physical parameters of the sound, such as {intensity(t) (real-time intensity value), frequency_distribution(t) (real-time frequency distribution), rhythm(bpm) (beat rate (beats per minute))}.

[0042] In the embodiments of this application, the text visual parameters include, but are not limited to, at least one of the following: font weight, font color, font animation effect, and font special effects.

[0043] In the embodiments of this application, the visual element parameters include, but are not limited to, at least one of the following: shape, speed, color, position, trajectory, amplitude, and animation effect.

[0044] In this embodiment, a configurable sound feature and visual parameter mapping rule base can be maintained. When the speaker's emotional sound features and / or ambient sound features are detected, the text visual parameters and / or visual element parameters corresponding to the emotional sound features and / or ambient sound features are searched from the sound feature and visual parameter mapping rule base.

[0045] It should be noted that the sound feature and visual parameter mapping rule base may include, but is not limited to, at least one of the following mapping rules: mapping rules between speaker's emotional sound features and text visual parameters, mapping rules between ambient sound features and text visual parameters, mapping rules between speaker's emotional sound features and visual element parameters, and mapping rules between ambient sound features and visual element parameters. The specific rules can be selected according to the actual situation, and this application embodiment does not impose specific limitations.

[0046] It should be noted that when detecting the speaker's emotional voice features, the corresponding text visual parameters and / or visual element parameters can be searched from the voice feature and visual parameter mapping rule base. In other words, the speaker's emotion can be reflected in the text subtitles and / or visual objects displayed in the video stream. When detecting ambient sound features, the corresponding text visual parameters and / or visual element parameters can be searched from the voice feature and visual parameter mapping rule base. In other words, ambient sound features can be reflected in the text subtitles and / or visual objects displayed in the video stream. However, if the video segment corresponding to the audio segment with detected ambient sound features does not have subtitles, the ambient sound features can be reflected only in the visual object display. The above is only one optional embodiment, and specific choices can be made according to actual conditions. This application does not impose specific limitations on the embodiments.

[0047] For example, taking the speaker's emotions as reflected in the text subtitles of a video stream as an example, when processing the narrator's (speaker's) subtitles, the processing is based on the narrator's emotional voice characteristics. For instance, when C_emotion is detected as "Excitement" and I_intensity is high, the engine will query the rule base to generate a set of visual parameter instructions for the corresponding text subtitles, such as {font weight: 900, font animation: enable pulse animation, animation speed: intensity parameter * k_1, font outer glow color: orange-red, enable flame particle effects}. Here, k_1 is an adjustable coefficient. Conversely, if C_emotion is "Sadness", the parameters might be {font color: light blue, font animation: slow fade-out, font animation duration: (1-I_intensity)*k_2}.

[0048] For example, taking the representation of ambient sound features in visual object display as an example, when processing ambient sound or sound effects, the ambient sound feature S_vector is the primary basis. The generated visual object is not necessarily bound to specific text. For instance, for a sound of type "Cheer," an independent visual object is generated, whose parameters are determined by A_params, such as {shape: wavy, origin: screen edge, propagation speed: intensity(t)*k_3, amplitude: intensity(t)*k_4}. For a sound of type "Heartbeat," a visual object of {shape: circle, color: red, position: center, animation: pulsating, animation beat: rhythm(bpm)} is generated, achieving synchronization with heart rate.

[0049] S103. Adjust the visual style of the subtitles in the video stream using text visual parameters and / or generate visual objects using visual element parameters, and composite the visually styled subtitles and / or visual objects into the video stream to obtain a video stream with visually styled subtitles and / or visual objects.

[0050] In this embodiment, text information is first identified from the audio stream; then, subtitles for the video stream are generated using the text information. Afterward, the visual style of the subtitles for the video stream is adjusted using text visual parameters.

[0051] In this embodiment of the application, based on text visual parameters and / or visual element parameters, graphics rendering technology is used to render subtitles and visual objects, and accurately overlay them onto the appropriate positions of video frames.

[0052] S104. Output a video stream with visually styled subtitles and / or visual objects to play the video stream with visually styled subtitles and / or visual objects on a playback interface.

[0053] In this embodiment, a video stream with visually styled subtitles and / or visual objects can be output to a media source, and the media source can transmit the video stream to the playback interface of a playback device for playback. Alternatively, a video stream with visually styled subtitles and / or visual objects can be output to the playback interface of a playback device for playback.

[0054] It should be noted that the video stream played in this application includes visually styled subtitles and / or visual objects, and these visually styled subtitles and / or visual objects are strongly correlated with and synchronized in real time with the speaker's emotional voice characteristics and / or ambient sound characteristics detected from the audio stream. For example, the frequency of the "pulsating" effect of the subtitles is highly synchronized with the energy fluctuations or speech rate changes of the narrator's voice; or the diffusion speed and amplitude of the "sound wave" graphic appearing on the screen match the loudness change curve of the audience's cheers in the background, significantly improving the fidelity and dimensionality of information transmission, creating an unprecedented immersive viewing experience for all viewers, especially those with hearing impairments.

[0055] Understandably, by detecting the speaker's emotional voice features and / or ambient sound features in the audio stream, the visual parameters of the text and / or the parameters of the visual elements are determined to adjust the visual style of the subtitles in the video stream and / or generate visual objects, which are then synthesized into the video stream. This allows the subtitles and / or visual objects in the video stream used for playback on the interface to accurately match the speaker's true emotional state and the atmosphere of the scene, significantly improving the fidelity and dimensionality of information transmission, thereby significantly improving the accuracy of emotion recognition.

[0056] Based on the above embodiments, this application proposes a system for generating emotional subtitles based on multimodal acoustic features. See [link to relevant documentation]. Figure 2 The system can be composed of an audio / video stream input module, a sound source separation and text transcription module, a speech emotion analysis module, a sound field feature extraction module, a dynamic visual effects generation and mapping engine, and a subtitle rendering and synthesis module. The audio / video stream input module serves as the system's standard input interface, responsible for receiving real-time or pre-recorded media streams containing multi-channel audio and video. The sound source separation and text transcription module receives the raw audio stream and performs sound source separation and speech recognition processes in parallel. The speech emotion analysis module is used to deeply analyze the speaker's emotional voice features based on the speech extracted from the sound source; the sound field feature extraction module is used to construct ambient sound features based on the environmental sound effects extracted from the sound source. The dynamic visual effects generation and mapping engine is the bridge connecting "perception" and "expression," serving as the decision-making and scheduling center of the entire system. It receives the text, speaker's emotional voice features, and ambient sound features generated by the sound source separation and text transcription module, the speech emotion analysis module, and the sound field feature extraction module, respectively. Combining these with a "feature-visual" mapping rule base, it activates different mapping logics based on the source and type of the sound to obtain text visual parameters and / or visual element parameters. The subtitle rendering and compositing module is a graphics processing unit that receives raw video frames and detailed visual text and / or visual element parameters generated by the dynamic visual effects generation and mapping engine. It executes these parameter instructions: for text subtitles, it handles text layout, font rendering, and applies specified dynamic effects (such as glow, bolding, and animation); for sound field visual objects, it generates and renders corresponding graphics (such as sound waves and pulsating dots). Finally, this module precisely composites all rendered visual elements onto the corresponding raw video frames, generating the final output video stream.

[0057] Based on the above embodiments, a method for generating emotional subtitles based on multimodal acoustic features is proposed, see [link to relevant documentation]. Figure 3 The method may include: 1. User / media source initiates processing, inputting the media stream into the audio / video stream input module.

[0058] It should be noted that the media stream is a composite media stream that includes audio and video tracks.

[0059] 2. The audio and video stream input module transmits the audio stream to the sound source separation and transcription module for processing.

[0060] It should be noted that the sound source separation and transcription module applies a sound source separation algorithm to decompose the mixed audio into multiple logically independent audio tracks, such as the narrator's voice and ambient sound. Simultaneously, this module performs ASR on the narrator's voice track to generate an initial text string. The separated audio tracks and text are then distributed to subsequent parallel processing modules.

[0061] It should be noted that 6 and 7 are the two parallel branches of parallel feature extraction.

[0062] 3. The sound source separation and transcription module separates the narration speech and transmits it to the speech emotion analysis module.

[0063] 4. The sound source separation and transcription module separates the ambient / effect sound and transmits it to the sound field feature extraction module.

[0064] 5. The sound source separation and transcription module transmits ASR text to the dynamic visual effects generation and mapping engine.

[0065] 6. The voice emotion analysis module extracts acoustic features from the narration and outputs the emotional voice features to the dynamic visual effects generation and mapping engine.

[0066] It should be noted that the voice sentiment analysis module receives the narrator's voice. The module analyzes acoustic features such as fundamental frequency, energy, and MFCC, and uses a built-in deep learning classifier to determine the sentiment category and intensity of the current voice segment in real time, ultimately generating an emotional voice feature E_vector.

[0067] 7. The sound field feature extraction module extracts physical parameters from the ambient / effect sound and outputs ambient sound features to the dynamic visual effects generation and mapping engine.

[0068] It should be noted that the sound field feature extraction module receives ambient sound or effects sound tracks. Depending on the sound type, different signal processing techniques are used to quantify its loudness, spectral characteristics, rhythm, and other physical parameters, ultimately generating an ambient sound feature S_vector.

[0069] 8. The dynamic visual effects generation and mapping engine combines text, emotional sound features, and ambient sound features to make decisions, and generates text visual parameters and / or visual element parameters according to the "feature-visual" mapping rule library.

[0070] It's important to note that the dynamic visual effects generation and mapping engine, acting as the central node, gathers all the information generated in the preceding steps: ASR text, E_vector, and S_vector. The engine's internal decision-making logic is activated, transforming the input E_vector and S_vector into specific, executable sets of visual style parameters based on a pre-defined rule base or model. For example, it determines the font, color, and animation effects for the text "Gooooal!", and the shape, speed, and trajectory of the textless sound field information "audience cheers" to be presented as visual symbols.

[0071] 9. The dynamic visual effects generation and mapping engine transmits text visual parameters and / or visual element parameters to the subtitle rendering and compositing module.

[0072] 10. The audio and video stream input module transmits the original video frames to the subtitle rendering and compositing module.

[0073] 11. The subtitle rendering and compositing module renders dynamic subtitles and visual elements, and composites them into the original video frames to obtain a composite video stream.

[0074] It should be noted that the subtitle rendering and compositing module receives a set of visual parameter instructions and the original video stream. Based on the instructions, this module uses graphics rendering technology to dynamically render text and graphic elements and accurately overlay them onto the appropriate positions in the video frames, ultimately compositing a video stream with emotional and atmospheric visual elements.

[0075] 12. The subtitle rendering and compositing module sends the composited video stream to the user / media source.

[0076] It should be noted that the above process is carried out in real time, ensuring close synchronization between visual effects and sound changes.

[0077] It is understood that this application proposes a multimodal parallel acoustic analysis based on sound source separation. Through the collaborative work of a "sound source separation and text transcription module," a "speech emotion analysis module," and a "sound field feature extraction module," it effectively distinguishes target speech from ambient sound in mixed audio tracks and differentiates and quantifies the acoustic characteristics of different sound sources (emotional prosody of speech and physical properties of ambient sound). This achieves a high degree of accuracy and dimensionality in emotion and atmosphere recognition.

[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to a specific application scenario. It should be noted that this embodiment is merely a preferred implementation of the invention and is not intended to limit the scope of protection of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0079] The application scenario of this embodiment is a hearing-impaired user watching a live broadcast of an important football match through an intelligent video live streaming platform equipped with the system proposed in this application embodiment. The system will convert the sound "texture" of the stadium into dynamic visual information in real time, providing the user with an immersive viewing experience.

[0080] Scene 1: Offensive organization and the creation of a tense atmosphere In a crucial moment of the second half, the home team launched a highly threatening fast counter-attack. At this moment, the system's audio and video stream input module received the live signal. The sound source separation and text transcription module immediately began working, successfully separating the commentary audio track, the background audio track of the fans, and the sound effects of the ball and players touching it. Simultaneously, it performed real-time speech recognition on the commentator's audio track, transcribing it into text: "The home team has launched a fast counter-attack! Three passes and two layups have already passed the midfield! This is an excellent opportunity!" Meanwhile, the separated commentator's voice is fed into the speech sentiment analysis module. This module analyzes the prosody and spectral features of the speech signal, detecting a continuous and stable increase in the commentator's fundamental frequency (pitch), a significantly faster speaking speed, and an energy output far exceeding their average level. The deep learning model within the module combines these features to output a clear emotional sound feature E_vector, with its emotion category C_emotion identified as "Tension," and a normalized value of 0.8 for the emotion intensity I_intensity. On the other hand, the separated ambient sound is fed into the sound field feature extraction module. This module analyzes that the loudness of the fans' cheers is increasing exponentially, with the spectral energy rapidly expanding from a low-frequency crowd buzz to a full-band including high-frequency shouts. Based on this, the module outputs an ambient sound feature S_vector, with its sound type T_sound defined as "CrescendoCheer," along with real-time changing loudness and spectral distribution parameters.

[0081] These two feature vectors, along with the transcribed text, are fed into a dynamic visual effects generation and mapping engine. Based on internal rules, the engine first processes the text captions. Since the emotion category is "tension," the engine instructs the visual style of the captions to change: the font edges begin to exhibit a subtle, rapid shaking effect, and the font color transitions from standard white to a warm yellow representing warning and passion. Next, the engine processes ambient sound features, generating semi-transparent red visual ripples representing the home team's jersey color at the left and right edges of the screen. The diffusion speed and opacity of these ripples are positively correlated with the loudness parameter in the sound field vector. As the fans' cheers increase, the ripples converge towards the center of the screen at an increasingly faster speed, visually creating a strong sense of pressure and anticipation. Although the user cannot hear the sound, they can clearly perceive the rapidly escalating tension in the stadium through the dynamic changes in the captions and the energy waves converging at the screen edges.

[0082] Scene 2: Shooting on goal and the outburst of emotions As the striker took the shot, the ball swished through the net. In that instant, the commentator roared at the top of his lungs, "Goooooal! Goal! Goal! Absolutely a perfect corner! A world-class goal!" The cheers of the fans in the stadium reached their peak at that moment.

[0083] In this scenario, the speech emotion analysis module detected that the commentator's voice energy had reached its maximum peak, the fundamental frequency had reached the upper limit of their personal vocal range, and it was accompanied by typical acoustic features of a highly excited state, such as "voice cracking." Based on this, the module outputs the emotional voice feature E_vector as (C_emotion="Excitement", I_intensity=0.99). The sound field feature extraction module also detected that the loudness of the ambient sound had reached its peak.

[0084] Upon receiving these extreme parameters, the dynamic visual effects generation and mapping engine immediately executed the highest-priority visual mapping rules. For the subtitle "Gooooal!", the engine instructed the subtitle rendering and compositing module to perform the following operations: the font was instantly bolded and enlarged to 150% of its original size, the color changed to a dazzling gold, and pulsed outward glow effects and flame particle effects bursting from the strokes of the text were added. The duration of the entire word on the screen was also extended accordingly to match the duration of the commentator's long syllable. For the cheers from the audience, the red visual ripples previously generated at the edge of the screen instantly accelerated and converged at the center of the screen, forming a violent visual "explosion" effect that briefly covered the entire screen before quickly dissipating. Through this multi-layered, high-intensity visual stimulation, the system vividly and powerfully conveyed the ecstasy of the goal and the explosive soundscape to the user.

[0085] It should be noted that this solution deeply analyzes and visualizes the emotional and atmospheric information in audio. For live sports broadcasts, it can amplify the passion of a goal, greatly enhancing the viewing experience and interactivity; for films and television dramas, it can enhance the emotional tension of key plot points, create a unique visual style, and thus improve the artistic expression of the content; for live music performances, it can convey the vibrant atmosphere of the event to online viewers. Furthermore, this technology provides hearing-impaired individuals with information compensation far exceeding that of traditional subtitles, meeting market demand for accessibility design.

[0086] This application provides an electronic device. For example... Figure 4 As shown, the electronic device 1 includes: Acquisition unit 10 is used to acquire media streams; the media streams consist of audio streams and video streams; The determining unit 11 is configured to determine text visual parameters and / or visual element parameters based on the emotional voice features and / or ambient voice features when the speaker's emotional voice features and / or ambient voice features are detected from the audio stream. The adjustment and / or generation unit 12 is used to adjust the visual style of the subtitles of the video stream using the text visual parameters and / or generate visual objects using the visual element parameters; Composition unit 13 is used to composite the visually styled subtitles and / or the visual objects into the video stream to obtain a video stream with visually styled subtitles and / or visual objects. Output unit 14 is used to output the video stream with visually styled subtitles and / or visual objects to play the video stream with visually styled subtitles and / or visual objects on the playback interface.

[0087] Optionally, the electronic device further includes: a separation unit and an analysis unit; The separation unit is used to separate the speaker's speech signal from the audio stream; The analysis unit is used to analyze the acoustic features used to represent emotions extracted from the speech signal using a deep learning classifier to obtain the speaker's emotion type and emotion intensity; and to construct the speaker's emotional voice features based on the speaker's emotion type and emotion intensity. The determining unit 11 is further configured to, when separating the ambient sound effect from the audio stream, determine the sound type corresponding to the ambient sound effect and extract physical parameters related to the sound type from the ambient sound effect, and construct the ambient sound feature based on the sound type and the physical parameters.

[0088] Optionally, the acoustic features consist of prosodic features and / or spectral features, and the electronic device further includes: a data acquisition unit and a recognition unit; The acquisition unit is used to acquire emotion-related patterns from the prosodic features and / or the spectral features using the convolutional neural network; The recognition unit is used to identify the speaker's emotion type and emotion intensity by learning the time-series dependencies of the emotion-related patterns using the long short-term memory network.

[0089] Optionally, the determining unit 11 is further configured to determine that the sound type is ambient sound, and to obtain physical parameters for characterizing the intensity of the ambient sound by analyzing the time spectrum of the ambient sound effect through short-time Fourier transform; and / or, to determine that the sound type is effect sound, and to extract the rhythm data and the instantaneous energy peak of each beat of the ambient sound effect by using autocorrelation function analysis or beat detection algorithm, and to determine the rhythm data and the instantaneous energy peak of each beat as physical parameters related to the effect sound.

[0090] Optionally, the electronic device further includes: a search unit; The search unit is used to search for the text visual parameters and / or visual element parameters corresponding to the emotional sound features and / or the ambient sound features from the sound feature and visual parameter mapping rule base.

[0091] Optionally, the text visual parameters include, but are not limited to, at least one of the following: font weight, font color, font animation effects, and font special effects; The visual element parameters include, but are not limited to, at least one of the following: shape, speed, color, position, trajectory, amplitude, and animation effect.

[0092] Optionally, the recognition unit is further configured to recognize text information from the audio stream; The generation unit is used to generate subtitles for the video stream using the text information.

[0093] This application provides an electronic device that acquires a media stream, which consists of an audio stream and a video stream. Upon detecting the speaker's emotional voice features and / or ambient sound features in the audio stream, text visual parameters and / or visual element parameters are determined based on the emotional voice features and / or ambient sound features. The visual style of the subtitles in the video stream is adjusted using the text visual parameters, and / or visual objects are generated using the visual element parameters. The visually styled subtitles and / or visual objects are then composited into the video stream to obtain a video stream with visually styled subtitles and / or visual objects. The video stream with visually styled subtitles and / or visual objects is output to be played on a playback interface. Therefore, the electronic device proposed in this embodiment determines the text visual parameters and / or visual element parameters by detecting the speaker's emotional voice features and / or ambient sound features in the audio stream. This allows for the adjustment of the visual style of the subtitles in the video stream and / or the generation of visual objects, which are then synthesized into the video stream. This ensures that the subtitles and / or visual objects in the video stream used for playback on the playback interface accurately match the speaker's true emotional state and the atmosphere of the scene, significantly improving the fidelity and dimensionality of information transmission, thereby significantly improving the accuracy of emotion recognition.

[0094] Figure 5 This is a schematic diagram of the composition structure of an electronic device 1 provided in an embodiment of this application. In practical applications, based on the same disclosed concept of the above embodiments, such as... Figure 5 As shown, the electronic device 1 in this embodiment includes: a processor 15, a memory 16, and a communication bus 17.

[0095] The processor 15 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not specifically limit it.

[0096] In this embodiment, the communication bus 17 is used to establish a connection between the processor 15 and the memory 16; when the processor 15 executes the running program stored in the memory 16, it implements the following video playback method: Acquire a media stream; the media stream consists of an audio stream and a video stream; upon detecting the speaker's emotional voice features and / or ambient sound features from the audio stream, determine text visual parameters and / or visual element parameters based on the emotional voice features and / or the ambient sound features; adjust the visual style of the subtitles in the video stream using the text visual parameters and / or generate visual objects using the visual element parameters, and composite the visually styled subtitles and / or the visual objects into the video stream to obtain a video stream with visually styled subtitles and / or visual objects; output the video stream with visually styled subtitles and / or visual objects to play the video stream with visually styled subtitles and / or visual objects on a playback interface.

[0097] Furthermore, the processor 15 is also configured to separate the speaker's speech signal from the audio stream; when extracting acoustic features for characterizing emotions from the speech signal, analyze the acoustic features using a deep learning classifier to obtain the speaker's emotion type and emotion intensity; construct the speaker's emotional voice features based on the speaker's emotion type and emotion intensity; and / or, when separating ambient sound effects from the audio stream, determine the sound type corresponding to the ambient sound effects and extract physical parameters related to the sound type from the ambient sound effects, and construct the ambient sound features based on the sound type and the physical parameters.

[0098] Furthermore, the acoustic features consist of prosodic features and / or spectral features, and the deep learning classifier is composed of a convolutional neural network and a long short-term memory network. The processor 15 is further configured to acquire emotion-related patterns from the prosodic features and / or the spectral features using the convolutional neural network; and to identify the speaker's emotion type and emotion intensity by learning the time-series dependence of the emotion-related patterns using the long short-term memory network.

[0099] Furthermore, the processor 15 is also configured to determine that the sound type is ambient sound, analyze the time spectrum of the ambient sound effect through short-time Fourier transform to obtain physical parameters for characterizing the intensity of the ambient sound; and / or, determine that the sound type is effect sound, extract the rhythm data of the ambient sound effect and the instantaneous energy peak of each beat using autocorrelation function analysis or beat detection algorithm, and determine the rhythm data and the instantaneous energy peak of each beat as physical parameters related to the effect sound.

[0100] Furthermore, the processor 15 is also used to search for the text visual parameters and / or visual element parameters corresponding to the emotional sound features and / or the ambient sound features from the sound feature and visual parameter mapping rule base.

[0101] Furthermore, the text visual parameters include, but are not limited to, at least one of the following: font weight, font color, font animation effect, and font special effects; The visual element parameters include, but are not limited to, at least one of the following: shape, speed, color, position, trajectory, amplitude, and animation effect.

[0102] Furthermore, the processor 15 is also configured to identify text information from the audio stream and generate subtitles for the video stream using the text information.

[0103] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in an electronic device. The computer program implements the video playback method described above.

[0104] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the video playback method described above.

[0105] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0107] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A video playback method, characterized in that, The method includes: Acquire media streams; the media streams consist of audio streams and video streams; If the speaker's emotional voice features and / or ambient voice features are detected from the audio stream, text visual parameters and / or visual element parameters are determined based on the emotional voice features and / or the ambient voice features. The visual style of the subtitles in the video stream is adjusted using the text visual parameters and / or a visual object is generated using the visual element parameters. The visually styled subtitles and / or the visual object are then combined into the video stream to obtain a video stream with visually styled subtitles and / or a visual object. Output the video stream with visually styled subtitles and / or visual objects to play the video stream with visually styled subtitles and / or visual objects on the playback interface.

2. The method according to claim 1, characterized in that, After acquiring the media stream and before determining the text visual parameters and / or visual element parameters based on the emotional sound features and / or the ambient sound features, the method further includes: The speaker's speech signal is separated from the audio stream; acoustic features for characterizing emotions are extracted from the speech signal, and the acoustic features are analyzed using a deep learning classifier to obtain the speaker's emotion type and emotion intensity; the speaker's emotional voice features are constructed based on the speaker's emotion type and emotion intensity. And / or, in the case of separating ambient sound effects from the audio stream, determining the sound type corresponding to the ambient sound effect and extracting physical parameters related to the sound type from the ambient sound effect, and constructing the ambient sound features based on the sound type and the physical parameters.

3. The method according to claim 2, characterized in that, The acoustic features consist of prosodic features and / or spectral features. The deep learning classifier consists of a convolutional neural network and a long short-term memory network. The analysis of the acoustic features using the deep learning classifier to obtain the speaker's emotion type and emotion intensity includes: The convolutional neural network is used to collect emotion-related patterns from the prosodic features and / or the spectral features; By utilizing the Long Short-Term Memory network to learn the time-series dependencies of the emotion-related patterns, the speaker's emotion type and intensity can be identified.

4. The method according to claim 2, characterized in that, The step of determining the sound type corresponding to the ambient sound effect and extracting physical parameters related to the sound type from the ambient sound effect includes: The sound type is determined to be ambient sound. The time spectrum of the ambient sound effect is analyzed by short-time Fourier transform to obtain physical parameters that characterize the intensity of the ambient sound. And / or, determine that the sound type is an effect sound, use autocorrelation function analysis or beat detection algorithm to extract the rhythm data of the environmental sound effect and the instantaneous energy peak of each beat, and determine the rhythm data and the instantaneous energy peak of each beat as physical parameters related to the effect sound.

5. The method according to claim 1, characterized in that, The step of determining text visual parameters and / or visual element parameters based on the emotional sound features and / or the ambient sound features includes: From the sound feature and visual parameter mapping rule base, search for the text visual parameters and / or visual element parameters corresponding to the emotional sound features and / or the ambient sound features.

6. The method according to claim 1, characterized in that, The text visual parameters include, but are not limited to, at least one of the following: font weight, font color, font animation effects, and font special effects; The visual element parameters include, but are not limited to, at least one of the following: shape, speed, color, position, trajectory, amplitude, and animation effect.

7. The method according to claim 1, characterized in that, Before adjusting the visual style of the subtitles in the video stream using the text visual parameters, the method further includes: Identify text information from the audio stream; The text information is used to generate subtitles for the video stream.

8. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a communication bus; when the processor executes the running program stored in the memory, it implements the method as described in any one of claims 1-7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images