Video subtitle generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a video subtitle generation method, which comprises the following steps of: performing down-sampling processing on video data, extracting appearance features and motion features of a video frame sequence, generating video features through a feature coding module, and generating subtitles according to the video features. And combining an emotion category library and an emotion vocabulary library to carry out emotion coding on the video features to generate emotion features. When a first video frame is processed, initial caption segments and caption features are generated by utilizing initial emotion guide intensity; and when subsequent video frames are processed, generating corresponding subtitle segments and subtitle features by combining the video features and the emotion features of the current video frame and the subtitle features of the previous video frame. By fusing the video features and the emotion features, the emotion clues in the video can be accurately captured, and the subtitle segments matched with the emotion content of the video are generated. The emotion association between the video frames is analyzed through an attention mechanism, so that the continuity and consistency of subtitle generation are ensured, and the accuracy of subtitles is improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a video subtitle generation method, device, equipment and storage medium. Background Art

[0002] In the financial industry, video data is increasingly used in customer service, marketing, training and education. With the rapid increase in the amount of video data, financial institutions are faced with the challenge of efficiently processing and analyzing video content. Currently, existing financial video processing technologies mainly focus on extracting basic information from videos, such as scene recognition and speech transcription, but lack effective processing capabilities for deep-level information such as emotional information and emotional guidance in videos.

[0003] Existing technologies have difficulty in achieving automatic and accurate emotional video subtitle generation, especially in financial videos, where a lot of content needs to be accurately interpreted based on emotions. For example, emotional reactions expressed during customer consultations and changes in market sentiment are important factors in decision support and customer experience evaluation.

[0004] However, existing subtitle generation technologies have the following deficiencies in emotion recognition and subtitle generation:

[0005] It is impossible to accurately capture the complex emotional clues expressed in the video, especially in financial business videos, where changes in customer emotions and market sentiment are difficult to accurately record and reflect.

[0006] Existing video subtitle generation relies on basic video features, but is unable to combine emotional features to generate subtitles that match the emotional content of the video. This results in the generated subtitles not matching the emotional state in the video, making it difficult to meet the demand for high-accuracy subtitles in the financial field.

[0007] When processing video frames, existing technologies cannot effectively analyze the emotional association between the current frame and the previous frame, which affects the coherence and emotional consistency of subtitle generation.

[0008] Therefore, the existing technology has significant technical defects in the generation of emotional video subtitles. Especially when processing large amounts of video data in the financial field, it is difficult to accurately generate subtitles that can reflect the emotions of the video content, which limits the emotional analysis and application of video data. Summary of the invention

[0009] The main purpose of the present invention is to provide a video subtitle generation method, device, equipment and storage medium, aiming to solve the technical problem that the existing technology cannot accurately identify and process complex emotional clues in the video, resulting in the generated subtitles not matching the emotional content of the video.

[0010] To achieve the above object, the present invention provides a video subtitle generation method, comprising:

[0011] Acquire video data, perform downsampling processing on the video data, and obtain a video frame sequence after the downsampling processing;

[0012] Extracting appearance features from the video frame sequence using a two-dimensional convolutional neural network, and extracting motion features from the video frame sequence using a three-dimensional convolutional neural network;

[0013] Encoding the appearance feature and the motion feature through a feature encoding module to generate a video feature;

[0014] Based on the emotion category library and the emotion vocabulary library, using the emotion feature encoding module to perform emotion encoding on the video features to generate emotion features;

[0015] When processing the first video frame, based on the preset initial emotion guidance strength, combined with the emotion feature and video feature of the first video frame, a predefined starting symbol is used to generate an initial subtitle segment and a corresponding subtitle feature;

[0016] When processing the video frames after the first video frame, the attention mechanism is used to analyze the emotional guidance strength of the current video frame based on the video features, emotional features of the current video frame and the subtitle features corresponding to the previous video frame;

[0017] Generate a subtitle segment of the current video frame and corresponding subtitle features based on the emotion guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame;

[0018] A weighted sum of a sentiment center cross entropy loss and a hierarchical sentiment classification loss is defined as a total loss function, and a loss value between a caption feature of each video frame and a target sentiment feature corresponding to each video frame is analyzed according to the total loss function;

[0019] Optimizing a complete subtitle generation model based on the loss value, wherein the complete subtitle generation model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, and a subtitle feature extraction module;

[0020] Generate target subtitles based on the optimized complete subtitle generation model.

[0021] Furthermore, to achieve the above object, the present invention provides a video subtitle generating device, comprising:

[0022] A data sampling module is used to acquire video data, perform downsampling processing on the video data, and obtain a video frame sequence after the downsampling processing;

[0023] A feature extraction module, used to extract appearance features from the video frame sequence using a two-dimensional convolutional neural network, and to extract motion features from the video frame sequence using a three-dimensional convolutional neural network;

[0024] A feature encoding module, used for encoding the appearance feature and the motion feature through the feature encoding module to generate a video feature;

[0025] An emotion feature encoding module is used to perform emotion encoding on the video features based on an emotion category library and an emotion vocabulary library using the emotion feature encoding module to generate emotion features;

[0026] A subtitle generation module is used to generate an initial subtitle segment and corresponding subtitle features using a predefined starting symbol based on a preset initial emotion guidance strength, combined with the emotion features and video features of the first video frame when processing the first video frame;

[0027] The emotion guidance analysis module is used to analyze the emotion guidance strength of the current video frame using an attention mechanism based on the video features and emotion features of the current video frame and the subtitle features corresponding to the previous video frame when processing the video frames after the first video frame;

[0028] A subtitle processing and feature extraction module, for generating a subtitle segment of a current video frame and corresponding subtitle features based on the emotional guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame;

[0029] A loss analysis module, used to define a weighted sum of a sentiment center cross entropy loss and a hierarchical sentiment classification loss as a total loss function, and analyze a loss value between a subtitle feature of each video frame and a target sentiment feature corresponding to each video frame according to the total loss function;

[0030] A model optimization module, used for optimizing a complete subtitle generation model based on the loss value, wherein the complete subtitle generation model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, and a subtitle feature extraction module;

[0031] The subtitle generation module is used to generate target subtitles based on the optimized complete subtitle generation model.

[0032] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a video subtitle generation program stored in the memory and executable on the processor, wherein the video subtitle generation program implements the steps of the video subtitle generation method as described above when executed by the processor.

[0033] Furthermore, to achieve the above object, the present invention also provides a computer-readable storage medium, on which a video subtitle generation program is stored, and when the video subtitle generation program is executed by a processor, the steps of the video subtitle generation method described above are implemented.

[0034] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a method for generating video subtitles. The method performs downsampling processing on video data, extracts appearance features and motion features in a video frame sequence, generates video features through a feature encoding module, and performs emotion encoding on the video features in combination with an emotion category library and an emotion vocabulary library to generate emotion features. When processing the first video frame, the initial emotion guidance strength is used to generate an initial subtitle segment and subtitle features; when processing subsequent video frames, the video features, emotion features of the current video frame, and the subtitle features of the previous video frame are combined to generate corresponding subtitle segments and subtitle features. The subtitle generation model is optimized by calculating the loss value, and the final subtitles are generated. The present invention can accurately capture the emotional clues in the video by fusing video features and emotion features, and generate subtitle segments that match the emotional content of the video. The emotional association between video frames is analyzed through an attention mechanism to ensure the coherence and consistency of subtitle generation and improve the accuracy of subtitles. The optimized subtitle generation model effectively reduces the need for manual intervention, and the accuracy and efficiency of automatically generated subtitles are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0036] Figure 1 A schematic diagram of an application environment of a method for generating video subtitles in an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for generating video subtitles according to the present invention;

[0038] Figure 3 A schematic diagram of functional modules of a preferred embodiment of a video subtitle generating device of the present invention;

[0039] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0040] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0042] The video subtitle generation method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can perform downsampling processing on the video data through the user end, extract the appearance features and motion features in the video frame sequence, generate video features through the feature encoding module, and perform emotion encoding on the video features in combination with the emotion category library and the emotion vocabulary library to generate emotion features. When processing the first video frame, the initial emotion guidance strength is used to generate the initial subtitle segment and subtitle features; when processing subsequent video frames, the video features, emotion features of the current video frame and the subtitle features of the previous video frame are combined to generate the corresponding subtitle segment and subtitle features. The subtitle generation model is optimized by calculating the loss value, and the final subtitle is generated. The present invention can accurately capture the emotional clues in the video by fusing the video features and the emotion features, and generate the subtitle segment that matches the emotional content of the video. The emotional association between the video frames is analyzed by the attention mechanism to ensure the coherence and consistency of the subtitle generation, and improve the accuracy of the subtitle. The optimized subtitle generation model effectively reduces the need for manual intervention, and the accuracy and efficiency of the automatic generation of subtitles are significantly improved. Among them, the user end can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0043] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for generating video subtitles provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0044] like Figure 2 As shown, the video subtitle generation method proposed by the present invention comprises the following steps:

[0045] S10, acquiring video data, performing downsampling processing on the video data, and obtaining a video frame sequence after the downsampling processing;

[0046] In this embodiment, one or more video data segments are obtained from an external device or an internal system. The video data may come from a camera, a monitoring device, an Internet video file, or may be called from a video library. The video data is usually a continuous sequence of frames, each frame of which is image data, and each video segment may contain multiple segments, each segment representing a continuous portion of the video. This is achieved by file reading or streaming media reception. For example, in a monitoring system in the financial field, video data may be obtained from a monitoring camera in real time, or may be loaded by reading a locally stored video file.

[0047] Downsampling refers to reducing the number of video frames and the resolution of each frame to reduce the amount of video data processing while retaining the key information of the video. Downsampling usually includes temporal downsampling (reducing the number of frames) and spatial downsampling (reducing image resolution). Temporal downsampling reduces the total number of frames by skipping or averaging multiple time frames, while spatial downsampling simplifies image details by reducing the pixel resolution of each frame.

[0048] In temporal downsampling, one frame can be retained every n frames to reduce the number of frames; in spatial downsampling, the pixel resolution of the frame can be adjusted, such as using image reduction technology to reduce the computational burden. For example, in the financial field, surveillance videos often need to be recorded for a long time. Through downsampling, redundant data can be reduced and the focus can be placed on the key period for financial risk identification.

[0049] The downsampled video frame sequence refers to a set of frames that have been optimized for temporal and spatial downsampling, containing the main time points and scene content of the video. This frame sequence is used for subsequent feature extraction and subtitle generation. At this stage, the video frame sequence has been compressed, but key visual information and temporal order are still retained.

[0050] The downsampled frame sequence can be saved in a buffer or array and passed as input data to the subsequent feature extraction module. Each video frame is still arranged in chronological order, but the number and size of frames are significantly reduced. In financial scenarios, such a processing process can ensure that important market changes or customer behaviors are not ignored due to excessive data volume.

[0051] Example: In the financial sector, video surveillance data is widely used in customer behavior analysis and market reaction capture. For example, banks deploy surveillance systems in customer service areas and use video to analyze customers' expressions and movements to determine their emotional reactions. Through downsampling, banks can extract key frames and key scenes from surveillance videos, reduce a large number of irrelevant static images, and provide data support for subsequent emotion recognition and customer service optimization. This not only reduces the system load, but also enables quick responses at critical moments, improving customer experience.

[0052] By downsampling the video data, the amount of video data can be effectively reduced, and the consumption of computing resources can be reduced, which is particularly suitable for large-scale video data processing scenarios. The downsampling process retains the key frames and main visual information of the video, ensuring the accuracy and efficiency of subsequent processing steps (such as sentiment analysis and subtitle generation).

[0053] S20, extracting appearance features from the video frame sequence using a two-dimensional convolutional neural network, and extracting motion features from the video frame sequence using a three-dimensional convolutional neural network;

[0054] In this embodiment, a two-dimensional convolutional neural network (2D-CNN) extracts spatial features of an image by performing a convolution operation on a two-dimensional plane. In this step, the 2D-CNN is used to process each frame of a video frame sequence and extract static appearance features in the frame. Appearance features include visual elements such as the shape, color, and scene information of an object, which are usually related to the object and background in the video frame.

[0055] The two-dimensional convolution operation scans each frame of the image through the convolution kernel, processes the local area in the image, and extracts feature maps at different levels. After each layer of convolution operation, it is usually accompanied by a pooling operation (such as maximum pooling or average pooling) to reduce the dimension and calculation amount of the feature map. The extracted feature map can then be input into the subsequent neural network layer for further processing and analysis. For example, in the customer service monitoring video of the bank, 2D-CNN can be used to extract the basic static features of the customer's face (such as expression changes) and the surrounding scene background.

[0056] Three-dimensional convolutional neural network (3D-CNN) is a network that extends convolution operations in the time dimension and can capture both spatial and temporal features in video frames. Through 3D convolution operations, 3D-CNN can not only extract the spatial features of each frame, but also capture motion information and time series changes in the video through cross-frame convolution operations. Motion features include dynamic changes such as the movement trajectory and behavior of objects.

[0057] The 3D convolution operation uses a three-dimensional convolution kernel to perform convolution along the time axis of the video frame sequence, covering multiple frames of image data. Each convolution operation can extract the change information (such as object movement) in consecutive frames and generate a feature map containing time information. These feature maps are processed by 3D pooling layers to reduce the number of features and retain key motion information. For example, in financial market monitoring videos, 3D-CNN can capture the continuous changes in customer actions, such as the dynamic characteristics of sitting, standing, etc., to help financial institutions analyze customer emotional fluctuations.

[0058] For example, a 2D CNN (such as ResNet pre-trained on ImageNet) is used to extract appearance features, and a 3D CNN (such as 3D-ResNeXt pre-trained on Kinetics) is used to extract motion features. Next, the appearance features and motion features are further encoded using the Transformer block to obtain the final video features.

[0059] F=[Linear(F a );Linear(F m )]

[0060] V=Transformerv (F)| {Q,K,V}:F

[0061] Among them, F a Appearance Features, extracted from video frames by a two-dimensional convolutional neural network (such as ResNet). This parameter represents the static visual features in the image, such as object appearance, color, etc.; F m Motion Features, which are extracted from video frame sequences through three-dimensional convolutional neural networks (such as 3D-ResNet). This is the modeling of time-varying information in the video, reflecting motion information and dynamic scenes; Linear (F a ) and Linear(F m ) indicates that the extracted appearance features and motion features are linearly transformed respectively; the symbol “[;]” indicates that the two linearly transformed features are concatenated together to form a joint feature vector F.

[0062] V is the output of the formula, which usually represents the output feature after passing through the Transformer network. This feature can be used for subsequent tasks, such as sentiment analysis, scene understanding, etc.; this represents the Transformer model applied to feature F. The subscript v represents a specific stage or layer of this Transformer module, which is specifically used to process a certain type of feature (such as video features). The Transformer model usually includes multiple self-attention layers and feedforward neural networks to capture long-distance dependencies in the input data. {Q, K, V} are the three important components of Transformer, namely Query, Key, and Value. Their meanings are as follows:

[0063] Q(Query): The query vector extracted from the input feature F, used to perform attention query on other features.

[0064] K(Key): The key vector extracted from the input features F, used to match the query vector Q.

[0065] V (Value): The value vector extracted from the input feature F, which represents the actual information contained in the input feature. The attention mechanism calculates the correlation between Q and K, and then combines V to generate a new feature representation.

[0066] Example description: In the customer service scenario of financial institutions, the customer's actions and expressions in the surveillance video are important bases for financial institutions to judge customer satisfaction and emotional state. By extracting static features such as customer expressions through 2D-CNN and dynamic features such as customer continuous action changes through 3D-CNN, financial institutions can analyze customer behavior and emotional changes in real time. For example, when a customer's actions show anxiety, the system can automatically remind staff to pay more attention to the customer, thereby improving service quality and reducing customer churn.

[0067] By using a two-dimensional convolutional neural network to extract static appearance features from a video frame sequence, and using a three-dimensional convolutional neural network to extract motion features across frames, it is possible to accurately capture static and dynamic information in the video. This multi-dimensional feature extraction method improves computational efficiency while ensuring processing accuracy, laying the foundation for subsequent video sentiment analysis and subtitle generation.

[0068] S30, encoding the appearance feature and the motion feature by a feature encoding module to generate a video feature;

[0069] In this embodiment, the feature encoding module is a module for fusing different features (such as appearance features and motion features). By taking appearance features (static features extracted by a two-dimensional convolutional neural network) and motion features (dynamic features extracted by a three-dimensional convolutional neural network) as input, the feature encoding module encodes these features using an encoding mechanism (such as based on an attention mechanism or other encoding networks) to generate a higher-dimensional feature representation. The purpose of encoding is to further combine and compress feature information on the basis of maintaining the original information to generate a more representative feature vector.

[0070] The feature encoding module usually adopts a multi-head self-attention model based on the attention mechanism (such as the Transformer module), which can capture the correlation and importance between appearance features and motion features through a weighted mechanism. When inputting features, appearance features and motion features enter the encoding network through linear transformation or feature mapping layers, and the network assigns weights and enhances features to each feature in turn through multiple attention layers. Finally, the encoder outputs these processed feature vectors as a higher-level joint feature representation. For example, in customer monitoring analysis, the system can combine the customer's static facial expressions and dynamic movement changes through the encoding module to generate a unified emotional feature for further analysis.

[0071] Video features are the final output results after being processed by the feature encoding module. They unify the appearance features and motion features. Video features not only contain the static information of the video (such as the shape and color of the scene and objects), but also incorporate the dynamic information in the video (such as the movement trajectory and behavior changes of objects). This feature can be used as input for subsequent steps (such as emotion recognition and subtitle generation) to process video emotion analysis tasks.

[0072] Through feature fusion, video features retain key information of appearance and motion features. The feature encoding module further compresses the dimension of the feature vector through a multi-layer perceptron (MLP) or linear mapping to generate a more compact and highly integrated feature representation. This video feature can represent the global information of the video and can be further combined and processed with other modules (such as the emotion feature encoding module). For example, in financial customer service scenarios, video features can be used to identify fluctuations in customer emotions and provide valuable data support.

[0073] Example description: In customer sentiment analysis in the financial sector, video surveillance data is an important source for analyzing customer emotional fluctuations. For example, banks can collect customer behavior and expression data in real time through surveillance cameras. Through the feature encoding module, the system combines the customer's static appearance features (such as expression changes) with dynamic action features (such as body movements) to generate complete video features. These video features can provide support for subsequent sentiment analysis, helping financial institutions determine whether customers have anxiety, transaction risks or dissatisfaction, thereby improving service quality and strengthening risk control.

[0074] By encoding the appearance features and motion features through the feature encoding module to generate video features, the static and dynamic information of the video can be effectively integrated to ensure that the key content of the video is retained. In addition, through the compression and fusion of the encoding module, the dimension of the features is reduced and the computational efficiency is improved.

[0075] S40, based on the emotion category library and the emotion vocabulary library, using an emotion feature encoding module to perform emotion encoding on the video feature to generate an emotion feature;

[0076] In this embodiment, the emotion category library and the emotion vocabulary library are two important resources for describing emotion types and vocabulary in the emotion analysis process. The emotion category library usually contains predefined emotion categories, such as "anger", "happy", "sad", etc., which are used to identify emotion classification in videos or texts. The emotion vocabulary library is a specific vocabulary set associated with each emotion category, which represents the detailed emotion expression under the specific emotion category, such as "happy", "frustrated", "excited" and other words. The purpose of this step is to assist the encoding and classification of emotion features through these emotion resource libraries.

[0077] In the sentiment analysis task, we first select the sentiment category related to the current video features from the sentiment category library, and extract the fine-grained sentiment vocabulary under each sentiment category from the sentiment vocabulary library. In this way, we can further refine the emotional expression in the video content. For example, in a customer service scenario, by analyzing the customer's facial expressions and body language, the system can identify the customer's emotional state and match it with the sentiment vocabulary library to further determine the customer's specific sentiment vocabulary.

[0078] The emotional feature encoding module is an encoding module for mapping video features to emotional categories. The module uses the emotional category library and the emotional vocabulary library to convert video features into emotional features through an encoding mechanism. The encoding module can be a network based on the attention mechanism or other encoding models, which identifies emotional features in the video by analyzing the similarity between video features and emotional categories.

[0079] After receiving the video features, the emotion feature encoding module first preprocesses the video features and matches them according to the predefined emotion categories in the emotion category library. The matching process usually uses similarity analysis or classification networks to generate a preliminary emotion category distribution by comparing the similarity between the video features and each emotion category. Then, the emotion feature encoding module further refines the specific emotion vocabulary under the emotion category through the emotion vocabulary library to make the emotion features more specific and accurate. The final output emotion features will contain the emotion information reflected in the video. For example, when a customer expresses anxiety at the bank service desk, the system maps the customer's expression features to the emotion category of "anxiety" through the encoding module.

[0080] Emotional features refer to the information related to emotion categories and emotional vocabulary extracted from video features by the emotion feature encoding module. These emotion features can represent the specific emotions in the video, including both static expression information and dynamic behavior information. Emotional features can be used in subsequent emotion classification, analysis, and subtitle generation tasks to ensure that the emotional expression is consistent with the video content.

[0081] By encoding the emotion of video features, the generated emotion features will contain emotion information in multiple dimensions, including the weight distribution of emotion categories and the matching results of specific emotion words. These features can be further input into downstream tasks (such as subtitle generation modules) to generate text content that matches the emotion.

[0082] For example, the Emotion Encoder introduces emotion priors through a tree-structured emotion learning scheme involving psychological categories (catalog level) and common emotion words (lexicon level). Assuming that the emotion category dictionary where c i represents the i-th category, N cRepresents the number of emotion categories. The video feature is V, and the emotion category is F c It can be encoded by the Transformer block as:

[0083]

[0084] F c It is the emotion category feature, which means that the Transformer model is used to extract the emotion sequence dictionary D from the video feature V. c The emotional category feature representation extracted from D c is a dictionary of emotional sequences, representing all possible emotional categories, used to guide the extraction of emotional features. V is a video feature, which may include appearance and motion features extracted from the video. The predicted emotional category distribution P c (D c |X) is to predict the distribution of emotional categories based on the emotional features extracted by Transformer. Where X is the input feature used to generate the emotional category distribution (such as video feature V or other joint features)

[0085] Design a sentiment masking attention mechanism to filter out irrelevant sentiment words on the sentiment tree and learn the sentiment category F c The sentiment word features are given by c and sentiment word dictionary Only consider the first K sentiment categories and select the sentiment words contained in these K categories to generate the sentiment mask 0-1 matrix Add mask A to V and D through the Transformer block w We can get the emotional feature representation Among them, E is the global emotional clue based on video features and tree structure, E0 is the initial state of the dynamic emotional perception path, and the w |D w The distribution of sentiment words on .

[0086]

[0087] E is the emotional feature representation, which represents the global emotional features extracted by Transformer.

[0088] Transformer w It is used to convert the sentiment dictionary D w And the video feature V is encoded into a Transformer model of emotional features.

[0089] Mask=A is an attention mask, which is used to control which sentiment words or sentiment information are focused on in the self-attention calculation.

[0090] Example: In a financial service center, cameras collect real-time videos of interactions between customers and staff. The system first downsamples and extracts features from the video to capture visual and auditory information such as the customer's facial expressions and tone changes. Then, the video content is emotionally encoded through the emotion category library and emotion vocabulary library to identify customer emotions (such as anxiety, anger, or satisfaction). The emotional features are further refined and processed through the attention mechanism to ensure that the emotional guidance accurately matches the video content.

[0091] When the system recognizes that a customer is anxious or dissatisfied, it can generate corresponding emotional subtitles on the screen to prompt the service staff, showing "the customer is anxious and may be dissatisfied with the current service content" to remind the service staff to adjust the communication method. At the same time, the system can automatically record this emotional information to provide data support for subsequent customer feedback or complaint management.

[0092] Financial institutions can improve customer experience invisibly. When customers are emotionally unstable, service personnel can adjust their words or provide more help in time to avoid conflicts or customer loss. This not only reduces the burden of manual analysis of emotions, but also improves service efficiency and quality. In this way, banks and other financial institutions can more effectively understand and respond to customers' emotional needs, enhance customer stickiness and improve service levels.

[0093] By using the emotion category library and emotion vocabulary library, and combining the emotion feature encoding module to perform emotion encoding on video features, the generated emotion features can express the emotion information in the video in detail and ensure the accuracy of emotion analysis. This not only improves the accuracy of emotion recognition, but also generates more emotionally expressive subtitles.

[0094] S50, when processing the first video frame, based on a preset initial emotion guidance strength, combined with the emotion feature and video feature of the first video frame, using a predefined start symbol to generate an initial subtitle segment and a corresponding subtitle feature;

[0095] In this embodiment, the initial emotion guidance strength is an emotion parameter predefined by the system for guiding subtitle generation. When processing the first video frame, since there is no previous subtitle segment for reference, the initial emotion direction of subtitle generation is determined by the preset emotion guidance strength. This parameter can be adjusted according to the specific scene, such as presetting the emotion guidance strength of "tension" or "excitement" in a movie clip. The preset initial emotion guidance strength can be trained from historical data through an emotion analysis model, or it can be customized by the user. Depending on the application scenario, for example in financial services, if the system initial emotion is set to "polite" or "positive", the content generated by the subtitles will be guided around these initial emotions.

[0096] Emotional features refer to the emotional information extracted from video features by the emotional feature encoding module, while video features are features containing scene and dynamic information generated by encoding the appearance features and motion features of the video frame sequence. These two features will be combined in the processing of the first video frame to guide subtitle generation. The system first extracts the emotional features and video features of the first video frame. Emotional features can be extracted from information such as the customer's expression and body language, while video features mainly contain objects, scenes, and motion changes in visual information. By combining these two types of features, the generated subtitles can accurately express the emotions and content in the video. For example, in a customer service scenario, the system combines the customer's emotional features (such as nervousness) and video features (such as the customer waving quickly) to generate appropriate subtitles, such as "The customer is anxious."

[0097] The predefined start symbol is a special marker that guides the first segment of subtitle generation. In the subtitle generation process, a start symbol is usually required as the beginning of the generation sequence to ensure that the subtitle generation proceeds smoothly. When processing the first video frame, the system will first insert a predefined start symbol (such as "[START]" or other specific identifiers), and then generate a subtitle segment based on emotional features and video features. The role of this start symbol is to ensure that the generation process has a clear starting point and provide a contextual connection for the subsequent subtitle generation process.

[0098] The initial subtitle segment is the first subtitle generated by the system when processing the first video frame. Subtitle features are feature information related to the subtitle segment, usually including semantic information and emotional information of the subtitle. According to the emotional guidance strength, video features and emotional features, the initial subtitle segment generated by the system can reflect the main content and emotion of the first video frame. At the same time, the system generates corresponding subtitle features to represent the specific attributes and emotional information of the subtitle segment. The subtitle features will be used for subtitle generation of subsequent frames to ensure that the subtitles are consistent with the video content and emotional expression.

[0099] By combining the preset initial emotion guidance strength, emotion features and video features to generate the initial subtitle segment when processing the first video frame, it can be ensured that the subtitle generation can accurately express the emotional content in the video from the very beginning and provide a strong foundation for subsequent subtitle generation.

[0100] S60, when processing video frames after the first video frame, using an attention mechanism to analyze the emotional guidance strength of the current video frame based on the video features and emotional features of the current video frame and the subtitle features corresponding to the previous video frame;

[0101] In this embodiment, the video features of the current video frame are appearance features and motion features extracted by a two-dimensional convolutional neural network and a three-dimensional convolutional neural network. The emotional features are emotional information extracted from the current video frame by an emotional feature encoding module, which is used to analyze the emotional state of the characters in the video. Whenever the system processes a new video frame, it extracts the appearance features (such as the customer's expression, clothing) and motion features (such as hand movements, posture changes) of the frame, and combines the emotional feature encoding module to generate the emotional features of the frame. These features will be used for emotional guidance intensity analysis in subsequent steps. For example, in financial services, if the customer in the video shows anxious emotions, the system will extract relevant emotional features and generate subtitles to prompt "The customer is nervous."

[0102] Subtitle features are the subtitle information generated by the previous video frame and its corresponding emotional and semantic features. These features will be used in the processing of the current video frame to ensure the continuity and contextual relevance of subtitle generation. When processing the current video frame, the system will refer to the subtitles and their features generated by the previous video frame (including the emotional and semantic information of the subtitle segments), so as to ensure that the logic and emotion of subtitle generation are consistent. For example, if the subtitles of the previous video frame indicate that the customer is asking a question, the system will continue to generate subtitle segments related to the question to avoid emotional jumps or semantic breaks between subtitles.

[0103] The attention mechanism is a method used to focus on important emotions and video features in the current frame, analyze the emotional content that needs to be emphasized in the current frame, and generate the corresponding emotional guidance strength. This strength will directly affect the accuracy of emotional expression when the subtitles of the current frame are generated. When processing the current video frame, the attention mechanism will focus on analyzing which features in the current frame are key factors in emotional guidance based on the features in the video frame (such as the customer's emotions, facial expressions, and movements) and the subtitle features of the previous video frame. Through weight distribution, the system will focus on processing the parts related to emotional expression to ensure that emotional information can be accurately conveyed when subtitles are generated. For example, if the customer's emotional expression in the current frame is relatively strong, the system will generate more emotional subtitles, such as "The customer seems very excited."

[0104] By combining the video features and emotional features of the current video frame with the subtitle features generated by the previous video frame, and analyzing the strength of emotional guidance through the attention mechanism, the system can accurately capture the emotional changes in the video and generate emotionally consistent subtitle clips in a timely manner.

[0105] S70, generating a subtitle segment of the current video frame and a corresponding subtitle feature based on the emotion guidance strength of the current video frame and the subtitle segment and subtitle feature generated by the previous video frame;

[0106] In this embodiment, the intensity of emotional guidance is determined by analyzing the emotional features, video features in the current video frame, and the subtitle features of the previous video frame. It indicates the degree of influence of the emotional information of the current video frame on the generation of subtitles. The intensity of emotional guidance is calculated through the attention mechanism. The system determines the intensity of emotional expression in the current frame based on the subtitle features of the previous video frame and the emotional features of the current video frame. For example, in a customer service scenario, the system generates the intensity of emotional guidance in the current frame by analyzing the changes in the customer's facial expressions, thereby determining the emotional color of the subtitles, such as "the customer's tone is slightly anxious."

[0107] The subtitle segment generated by the previous video frame contains the subtitle information of the previous frame, while the subtitle features describe the emotional and semantic characteristics of the subtitle segment. They are used to help generate subtitles for the current frame to maintain the continuity and contextual relevance of the subtitle content. When processing the current video frame, the system will call the subtitle segment generated by the previous video frame and its emotional features to ensure the emotional continuity of subtitle generation. For example, if the subtitle of the previous frame shows "Customer asks questions", the subtitle generated by the current frame will continue to ensure the smooth expression logic, such as "The customer further questions the specific details."

[0108] The subtitle segment of the current frame is generated based on the current emotional guidance strength and the subtitle features of the previous video frame. The subtitle segment refers to a specific text expression, and the subtitle feature contains the emotional attributes of the subtitle segment. The system first calculates the emotional guidance strength of the current frame through the attention mechanism, and then combines the subtitle segment and subtitle features of the previous frame to generate the subtitle content of the current frame. These subtitle segments not only reflect the semantic content of the current frame, but also reflect the emotional changes of the current frame. For example, when it is detected that the customer's emotions fluctuate violently, the system may generate a subtitle segment such as "The customer is emotional and speaks faster" and give it relevant emotional features.

[0109] By combining the emotional guidance strength of the current video frame and the subtitle segment and its features generated by the previous video frame, the system can effectively generate subtitles that are consistent with emotional changes and contextual coherence. This makes the subtitle content more emotionally expressive, especially in customer service scenarios, which can timely reflect the customer's emotional state, help service personnel understand customer emotions, and improve the interactive quality and efficiency of services.

[0110] S80, defining a weighted sum of a sentiment center cross entropy loss and a hierarchical sentiment classification loss as a total loss function, and analyzing a loss value between a subtitle feature of each video frame and a target sentiment feature corresponding to each video frame according to the total loss function;

[0111] In this embodiment, the emotion-centered cross entropy loss is a loss function used to measure the deviation between the emotion words in the subtitles generated by the system and the real emotion words. Cross entropy loss is often used in classification problems to measure the difference between the generated probability distribution and the real distribution. In this context, it is specifically optimized for emotion words to ensure that the emotional expression in the subtitles matches the real emotions in the video. The system compares the emotion words in the subtitle clips generated by the subtitle generation model with the target emotion words reflected by the emotional features of the video. The role of the emotion-centered cross entropy loss is to increase the loss value when the generated emotion words deviate from the actual emotion, forcing the model to make more detailed adjustments to the generation of emotion words during the training process. For example, in the financial field, when a customer shows uneasiness, if the subtitle generation system mistakenly generates an incorrect emotion expression such as "the customer happily expressed", the system will correct this deviation through the emotion-centered cross entropy loss.

[0112] Hierarchical sentiment classification loss is another loss function that is used to perform hierarchical optimization on the sentiment classification of videos. This loss is based on the sentiment features of the video and the sentiment categories generated by the subtitles, helping the model to better distinguish different levels of emotional expressions, such as weak, strong, or complex emotional states. Hierarchical sentiment classification loss divides emotions into different levels of categories, such as "anxiety", "anger", "doubt", etc., based on the emotional features in the video, and calculates the difference between the generated subtitles and the target sentiment categories in a hierarchical manner, ensuring that the system can more accurately identify and generate multi-level emotional expressions. In financial services, assuming that the customer's emotions gradually change from mild anxiety to strong dissatisfaction, the hierarchical sentiment classification loss will help the model gradually adjust the emotional expression so that the subtitles can accurately capture this emotional transition.

[0113] The total loss function is a weighted sum of the sentiment center cross entropy loss and the hierarchical sentiment classification loss, which is used to optimize the model's sentiment vocabulary generation and sentiment category classification capabilities at the same time. The weighted sum means assigning different weights to each loss to ensure that the system is fully optimized in both sentiment vocabulary generation and sentiment classification. During the training process, the system generates a total loss function by adding the sentiment center cross entropy loss and the hierarchical sentiment classification loss with certain weights. This total loss function is used to optimize the model so that it can both generate vocabulary that matches the emotion of the video and accurately classify the emotional state in the video. The adjustment of weights depends on the specific scenario requirements. For example, in a customer service scenario, if more precise sentiment classification is required, the weight of the hierarchical sentiment classification loss may be set higher.

[0114] The loss value refers to the error between the generated subtitle features and the target emotional features. By calculating the loss value between the subtitle features and the target emotional features of each video frame, the system can continuously adjust the emotional expression of the generated subtitles to make it more consistent with the actual emotions in the video. Every time the system processes a video frame, it calculates the loss value between the subtitle features generated by that frame and the target emotional features. This loss value is used to guide the model to adjust the emotional expression of the generated subtitles so that subsequent subtitle generation is more consistent with the emotional expression of the actual video. For example, if the customer's emotional expression in the video gradually becomes stronger, the system will adjust the generated subtitles in a timely manner through the analysis of the loss value to ensure that more accurate emotional vocabulary is used in the subtitles to reflect this change.

[0115] By defining the weighted sum of the emotion-centered cross entropy loss and the hierarchical sentiment classification loss as the total loss function, the system can accurately optimize the emotional expression in subtitle generation. It can not only generate subtitle vocabulary that conforms to the emotional context, but also effectively classify the emotional levels, ensuring the emotional accuracy and multi-level nature of the subtitle expression.

[0116] S90, optimizing a complete subtitle generation model based on the loss value, wherein the complete subtitle generation model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, and a subtitle feature extraction module;

[0117] In this embodiment, the loss value is used to measure the performance of the model in generating subtitles. By calculating the loss between the subtitle features generated by each video frame and the target emotional features, the model can adjust its parameters to optimize the effect of generating subtitles. By continuously reducing the loss value, the model can gradually improve the accuracy of the generated subtitles and the fit of the emotional expression. During the model training process, the loss value is calculated for each video frame processed. The loss value measures the degree of match between the generated subtitles and the video emotions through the emotional center cross entropy loss and the hierarchical emotional classification loss. The model uses the back propagation algorithm to adjust the weights of the neural network based on the size of the loss value, so that the model gradually learns how to accurately generate subtitles that match the video content and emotions. The training process is repeated until the model loss value reaches the lowest point to ensure the optimal generation ability of the model.

[0118] 2D CNN is used to extract static appearance features such as scenes, objects, colors, etc. in a sequence of video frames. These features help to understand the visual content of the video and provide a basis for sentiment analysis and subtitle generation. The 2D convolutional neural network performs convolution and pooling operations on each frame to extract appearance features from the video frame. The extracted appearance features will be used as input for the subsequent encoding step to ensure that the model can capture the static visual information in the video.

[0119] 3D CNN is used to extract cross-frame motion features and analyze the dynamic changes and motion trajectories of objects in a video frame sequence. This is crucial for understanding the timing information and dynamic content in a video. The three-dimensional convolutional neural network performs convolution operations on the video frame sequence in the spatial and temporal dimensions to extract cross-frame motion information. These motion features are used together with appearance features in subsequent feature fusion and encoding steps.

[0120] The feature encoding module encodes two-dimensional and three-dimensional features to generate a joint feature representation. These encoded features contain static and dynamic information of the video and can more accurately represent the overall content of the video. This module uses an attention-based encoding method to fuse appearance features with motion features and capture the dependencies between them. The generated joint feature representation serves as the basis for subsequent emotion encoding and subtitle generation.

[0121] The emotion feature encoding module generates emotion features that can reflect the emotion clues of the video by encoding the emotion features of the video, combining the emotion category library and the emotion vocabulary library. The emotion feature encoding module first selects candidate emotion categories from the emotion category library, and then further refines the emotion expression through the emotion vocabulary library. The finally generated emotion features are used to guide the emotion expression in the subtitle generation process.

[0122] The subtitle feature extraction module is used to extract subtitle features from each generated subtitle segment. These features help the model associate the previously generated subtitle content with the subsequent subtitles to ensure the coherence and accuracy of the subtitles. The subtitle feature extraction module extracts features from the generated subtitle segments and converts them into feature vectors that can be used for subsequent processing. These subtitle features are used together with video features and emotional features to generate subsequent subtitles.

[0123] Through the process of optimizing the model based on the loss value, the system can gradually improve the accuracy of the generated subtitles and the precision of emotional expression. The modules work together to ensure that the static and dynamic content of the video, as well as the complex emotional clues, can be fully reflected in the subtitles.

[0124] S100, generating target subtitles based on the optimized complete subtitle generation model.

[0125] In this embodiment, in the previous steps, the model has adjusted the parameters of each module by training and optimizing the input video data multiple times, so that the generated subtitles are more accurate and can reflect the video content and its emotional characteristics. The optimized complete model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, a subtitle feature extraction module, etc. These modules work together to generate subtitles that conform to the video content and emotional characteristics.

[0126] The new input video data will be downsampled to generate a video frame sequence, and its appearance features and motion features will be extracted. The appearance features and motion features are fused through the feature encoding module to generate video features. At the same time, the emotional feature encoding module is used to emotionally encode the video features to generate emotional features. Based on the video features, emotional features and the subtitles generated by the previous video frame, the model analyzes the emotional guidance strength through the attention mechanism and generates the subtitles of the current video frame.

[0127] The final subtitle clip will combine the video content and emotional clues to ensure the consistency and coherence of the subtitles in content and emotional expression.

[0128] The newly input video data is downsampled to generate a downsampled video frame sequence. A two-dimensional convolutional neural network is used to extract the appearance features in the video frame, and a three-dimensional convolutional neural network is used to extract the motion features across frames. The appearance features and motion features are input into the feature encoding module, and the features are fused through the attention mechanism to generate video features containing static information and dynamic information. The emotional feature encoding module is used to perform emotional encoding on the video features to generate emotional features.

[0129] When processing each video frame, based on the emotional guidance strength of the current video frame, combined with the subtitle segment and subtitle features of the previous video frame, a long short-term memory neural network based on the attention mechanism is used to generate the subtitle segment of the current video frame. The final generated subtitle segment will be output according to the video content and emotional features, and will be highly consistent with the video.

[0130] Through the above steps, the optimized model makes subtitle generation more accurate, can accurately capture the emotional changes in the video, and generate subtitles that are closely related to the video content and emotions. This subtitle generation method reduces the workload of manual subtitle annotation, improves the automation level of video processing, and facilitates multimodal retrieval and sentiment analysis.

[0131] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a method for generating video subtitles. The method comprises the following steps: performing downsampling processing on video data, extracting appearance features and motion features in a video frame sequence, generating video features through a feature encoding module, and performing emotion encoding on the video features in combination with an emotion category library and an emotion vocabulary library to generate emotion features. When processing the first video frame, the initial emotion guidance strength is used to generate an initial subtitle segment and subtitle features; when processing subsequent video frames, the video features, emotion features of the current video frame, and the subtitle features of the previous video frame are combined to generate corresponding subtitle segments and subtitle features. The subtitle generation model is optimized by calculating the loss value, and the final subtitles are generated. The present invention can accurately capture the emotional clues in the video by fusing the video features and the emotion features, and generate subtitle segments that match the emotional content of the video. The emotional association between video frames is analyzed through an attention mechanism to ensure the coherence and consistency of subtitle generation, thereby improving the accuracy of subtitles. The optimized subtitle generation model effectively reduces the need for manual intervention, and the accuracy and efficiency of automatically generated subtitles are significantly improved.

[0132] In one embodiment, in the above S60, based on the video features and emotional features of the current video frame and the subtitle features corresponding to the previous video frame, an attention mechanism is used to analyze the emotional guidance strength of the current video frame, including:

[0133] S601, projecting the video features of the current video frame and the subtitle features corresponding to the previous video frame into the same embedding space through linear transformation;

[0134] S602, performing a weighted pooling operation on the projected video features and subtitle features to generate extended features;

[0135] S603, analyzing a cross-modal correlation score between the extended feature and the emotional feature;

[0136] S604, aligning the extended feature and the sentiment feature based on the cross-modal correlation score to form an aligned joint feature;

[0137] S605, dividing the joint feature into multiple subspaces;

[0138] S606, using the attention mechanism to capture the semantic information of each subspace and assigning a weight to each subspace;

[0139] S607: Based on the weights, analyze the emotion guidance strength of the current video frame according to the semantic information of the weighted subspace.

[0140] In this embodiment, the video features and subtitle features are in different feature spaces, and they must be mapped to the same embedding space through linear transformation, so that data of different modalities can be compared and calculated in the same dimension. By designing a linear mapping matrix, the features of each video frame and the previous subtitle features are projected into the same embedding space respectively to ensure that their dimensions are consistent. The purpose of this is to be able to directly perform feature fusion in the subsequent steps.

[0141] The projected features may still contain redundant information or low-correlation information, so through the weighted pooling operation, important features are given higher weights to generate more representative extended features. Through the weighted pooling operation, the projected features are processed separately and a set of comprehensive features are generated. This process uses the pooling layer to sort the importance of each feature, weight it according to its relevance to emotion, and filter out noise data.

[0142] This step aims to evaluate the correlation between extended features and emotional features to determine their consistency across different modalities (video and emotion) and ensure that the strength of emotional guidance is consistent with the video content. Using the cross-modal correlation analysis algorithm, the similarity score between extended features and emotional features is calculated. This score is used for the calculation of emotional guidance strength in the subsequent steps to ensure that emotional elements are taken into account when generating subtitles.

[0143] The features of different modalities are aligned using the previous correlation scores. The purpose of alignment is to ensure that the extended features and sentiment features are consistent in expressing the same semantic information. The alignment algorithm adjusts the extended features and sentiment features according to the correlation scores to generate joint features. The aligned joint features combine information from both video and sentiment, making them suitable for further analysis.

[0144] Dividing the joint features into multiple subspaces allows for a more fine-grained analysis of multi-level semantic information in video frames. These subspaces help capture different emotional expressions in the video. The joint features are decomposed into multiple subspaces using a subspace partitioning algorithm, each of which corresponds to a specific semantic dimension or emotional clue, thereby decomposing complex emotional information.

[0145] The attention mechanism is used to automatically focus on more important subspaces and assign higher weights to them, ensuring that the most relevant semantic information is used first in the subtitle generation process. By designing a multi-head attention mechanism, the features of each subspace are scanned and weights are assigned. These weights represent the importance of each subspace and are used to guide the focus on emotional features during the generation process.

[0146] By analyzing the weighted subspace semantic information, the emotional guidance strength of the current video frame is calculated. This strength determines the emphasis on emotional expression when generating subtitles at the current time point. Using the weighted semantic information, the emotional expression strength of the video frame is analyzed through the emotional guidance strength calculation model. The final output guidance strength will be used as a key parameter for subsequent subtitle generation.

[0147] For example, the weighted pooling method is used for the historically generated subtitle features and video combination features to effectively capture the changes in the current generation state. The video feature V is projected onto the subtitle feature V that is consistent with the historically generated subtitle feature V. In the same embedding space, we combine them to get the combined feature C, where W v is a learnable parameter and is a connection operation:

[0148]

[0149] in, represents the historically generated subtitle features, and represents the subtitle feature vector generated at time t-1. t-1 represents the time step, d t Represents the feature dimension associated with the time step, which means that these features change at each time step and capture the time-related dynamic features in the subtitle generation process.

[0150] The combined feature C is obtained by combining the video feature V with the historical subtitle feature Y t-1 The video features are formed by the weight matrix W v Converted into features related to the time step and combined with the subtitle features. The concatenated C is a 2(t-1)×d t dimensional matrix, capturing the features of historical subtitles and the current video frame, which may change dynamically over time steps.

[0151] Use weighted pooling to reconstruct the combined features into the same embedding space. is a learnable parameter, and M is a hyperparameter that controls the size of the feature space.

[0152]

[0153] Compared with the original features, the extended features not only include simple subtitle status, but also consider the emotional changes of video content, providing accurate support for emotional evolution at the element level.

[0154] After obtaining the subtitle features, the element-level emotional evolution is constructed by selecting the emotional sub-features that need to be reorganized. Human emotions are complex and changeable, and different subtitle generation stages should have different emotional focuses. Therefore, it is necessary to eliminate the negative impact of irrelevant emotional factors. In order to select irrelevant emotional elements, the cross-modal correlation is first calculated to align the evolved emotional features. With extended features The aligned extended features and sentiment features are then concatenated and projected into an n-vector to obtain the dynamic sentiment feature R be . Where W c and [;] are learnable parameters and connection operations, respectively. is the learnable weight value, ⊙ is the element-wise multiplication. Finally, using R be Get the emotional characteristics of elemental evolution

[0155]

[0156] The model is based on the emotional element E t-1 Dynamic emotional factors R be Based on the correlation of the two, the sentiment elements are selectively reorganized to focus on the most relevant sentiment aspects at each time step.

[0157] Feature Space Can be divided into different subspaces Representation, contains different semantic information and captures subtle emotional changes. Therefore, given the element-level evolved emotional features and the list of subspace numbers {k j |j=1,2,…,N k}, where N k represents the number of subspaces. For each k j ,Will Divide into

[0158]

[0159] Then, the semantic information of each subspace is captured through the attention mechanism. In this process, it is assumed that is the number of attention heads. Usually each attention head will output an N×k j Therefore, if there are k j Such heads can be concatenated to form a A three-dimensional tensor of:

[0160]

[0161] ⊙ is the element-wise multiplication, which can be formalized as:

[0162]

[0163] Represents the sentiment feature of the j-th subspace at time step t. Resh_ape stands for "reshape" and "addpositional encoding", which is The results are rearranged and positional encodings are added to generate New representation of .

[0164] Therefore, the semantics of different subspaces are reorganized to different degrees. In order to merge the semantic information of different subspaces to generate new sentiment features, another kind of attention is used to generate each subspace Weight:

[0165]

[0166]

[0167] is the attention weight, which represents the emotional features of the previous time step and subtitle features The attention score calculated by the attention mechanism.

[0168] Atten w is the self-attention mechanism, using As query Q, subtitle features As key K and value V, calculate the attention weight

[0169] R aft The attention weight After the pooling operation, the subspace weight coefficients are obtained by applying the Softmax function. Softmax normalizes the weights so that each subspace is appropriately weighted.

[0170] Finally, a weighted value is applied to each reorganized subspace feature, YesR aft The j-th element of generates the evolved sentiment feature:

[0171]

[0172] Represented by the sentiment features of all subspaces According to the weighting factor R aft,j Perform weighted summation and then add the sentiment feature E of the previous time step t-1, and get the updated sentiment feature E t .

[0173] By capturing the emotional changes at each generation step, the dynamic emotion-aware pathway ensures that the generated captions are emotionally relevant and synchronized with the video content. Furthermore, the multi-step nature of the emotion evolution module enables our approach to perform a fine-grained analysis of the emotional dynamics, thus enabling the generation of captions that accurately reflect the emotional nuances of the video content.

[0174] This embodiment can provide accurate emotional description in the multimodal video subtitle generation task through this dual-path collaborative generation network model, which not only improves the accuracy of subtitle generation, but also reduces the cost of manual annotation in cross-modal retrieval tasks.

[0175] In one embodiment, the above S70 includes:

[0176] S701, inputting the emotion guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame into a long short-term memory neural network based on the attention mechanism;

[0177] S702, weighting the input emotional guidance strength, the subtitle segment generated by the previous video frame, and the corresponding subtitle features;

[0178] S703, based on the weighted emotion guidance strength, subtitle segment and corresponding subtitle feature, generate the subtitle segment and corresponding subtitle feature of the current video frame through a long short-term memory neural network.

[0179] In this embodiment, the emotional guidance strength is combined with the subtitle information of the previous frame and input into the model. The emotional guidance strength represents the emotional expression strength of the current frame, and the subtitle segment and subtitle feature represent the encoded subtitle generated by the previous frame. The emotional guidance strength of the current video frame is input into the long short-term memory (LSTM) neural network together with the subtitle segment and subtitle feature of the previous frame, ensuring that the system can consider the emotional changes of the current frame and the historical subtitle information at the same time, so as to generate new subtitles for the current frame.

[0180] Through weighted processing, different input information has different importance when generating subtitles. The weights of emotional guidance strength, subtitle segments and subtitle features can be adjusted dynamically to ensure that the subtitle generation process of the current frame is more reasonable. A weighting mechanism is designed to give different weights to the input emotional guidance strength, subtitle segments and subtitle features. The weight values ​​are determined by the attention mechanism. According to these weights, the LSTM network focuses on processing more important information. For example, when the emotional intensity is high, it may pay more attention to emotional features, while when the emotional intensity is weak, it may pay more attention to subtitle features.

[0181] The LSTM network generates new subtitle segments and corresponding subtitle features through weighted information. LSTM can retain and update previous states, so it can consider previous subtitle history when generating new subtitles. The LSTM network generates subtitle segments for the current video frame through weighted input, and extracts features from the generated subtitle segments to obtain corresponding subtitle features. This process combines historical information with the emotional guidance strength of the current frame to ensure the coherence and emotional consistency of subtitle generation.

[0182] In this embodiment, through the application of such weighted input processing and long short-term memory neural network, the system can generate more coherent subtitles, and the subtitles can accurately reflect the emotional information of the current frame.

[0183] In one embodiment, the above S20 includes:

[0184] S201, performing normalization or standardization processing on the video frame sequence;

[0185] S202, inputting the processed video frame sequence into a two-dimensional convolutional neural network frame by frame, performing layer-by-layer convolution and pooling operations on each frame image to extract static appearance features from each frame image;

[0186] S203, inputting the processed video frame sequence into a three-dimensional convolutional neural network, performing three-dimensional convolution and pooling operations in spatial and temporal dimensions, so as to extract cross-frame motion features from the video frame sequence, wherein the motion features include dynamic change information and object motion trajectory in the video.

[0187] In this embodiment, normalization or standardization is a common step in data preprocessing, the purpose of which is to ensure that the input data is calculated under the same range or standard to avoid the adverse effect of excessive differences in eigenvalues ​​on the model. Normalization is to compress the data to a fixed interval (usually [0,1]), and standardization is to convert the data into a standard normal distribution with a mean of 0 and a standard deviation of 1. The pixel values ​​in the video frame are normalized or standardized, and min-max normalization or z-score standardization can be used to ensure that the pixel values ​​of each frame are compressed or standardized to ensure that the subsequent neural network calculations are more efficient.

[0188] Two-dimensional convolutional neural network (2D-CNN) is a neural network architecture for processing images. By processing video frames frame by frame, static features in the image can be extracted. Layer-by-layer convolution and pooling operations can gradually reduce the dimension of the feature map while maintaining image details and extract high-level appearance information. The processed video frame sequence is input into 2D-CNN frame by frame in chronological order. Each frame of video image extracts feature information in the image through the convolution layer, and then the dimension is reduced through the pooling layer to retain important feature information. In the specific implementation, the convolution kernel is responsible for capturing features such as the edge and shape of the image, and the pooling operation further reduces the size of the feature map to reduce computational complexity.

[0189] Static appearance features refer to the static content of each frame in the video, such as objects, backgrounds, colors, and other information. 2D-CNN can extract these time-invariant features to help the model recognize the basic content in the video. After multiple layers of convolution and pooling operations, 2D-CNN generates multi-dimensional feature maps. These feature maps contain scene, object, and color information in the video frame, representing the static appearance of the video. The number of channels in the feature map represents different feature patterns, such as texture, edges, etc.

[0190] Three-dimensional convolutional neural networks (3D-CNN) can not only process single-frame images, but also process dynamic information of multiple consecutive frames. 3D-CNN can extract cross-frame motion information from a video frame sequence by convolving the spatial and temporal dimensions. The processed video frame sequence is input into the 3D-CNN, which uses a three-dimensional convolution kernel to perform convolution operations in the temporal and spatial dimensions. Each convolution operation processes a group of consecutive video frames to extract the temporal variation characteristics of the video, especially the motion trajectory of objects in the temporal dimension. Then, through the three-dimensional pooling operation, the dimension of the feature map is reduced to retain important motion information.

[0191] Motion features reflect the motion information and dynamic changes of objects in the video. 3D-CNN can capture the motion trajectory of objects and the dynamic changes of scenes over time in the video. Through 3D convolution and pooling operations, 3D-CNN generates feature maps containing motion features, reflecting the position changes and trajectories of objects in the time dimension. These motion features are crucial for describing the dynamic content of the video, such as object movement, behavior, etc.

[0192] By combining 2D-CNN and 3D-CNN, this embodiment can not only capture static information in the video, but also extract dynamic motion features to ensure a comprehensive understanding of the video content.

[0193] In one embodiment, the above S30 includes:

[0194] S301, inputting the appearance feature and the motion feature into a feature encoding module;

[0195] S302, the feature encoding module uses an encoding method based on an attention mechanism to capture the dependency between appearance features and motion features;

[0196] S303, based on the dependency relationship, feature fusion is performed on the appearance feature and the motion feature through a feature encoding module to generate a joint feature representation, where the joint feature representation includes static information and dynamic information of the video;

[0197] S304, compressing and fusing the joint features through a multi-layer perceptron or linear mapping to generate final video features.

[0198] In this embodiment, after the appearance features and motion features of the video frame sequence are extracted, these features are input into the feature encoding module for processing. The feature encoding module is a neural network module for feature fusion and encoding. The feature encoding module receives feature data output from the two-dimensional convolutional neural network and the three-dimensional convolutional neural network. The appearance features represent the static visual information of the video frame, and the motion features represent the dynamic information that changes over time. These features are fed into the encoding module as input and enter the next step of fusion and encoding process.

[0199] The encoding method based on the attention mechanism can dynamically adjust the weights between features, especially between appearance features and motion features. The attention mechanism captures the dependencies between different features by calculating the correlation between them, ensuring that the static and dynamic information of the video can be accurately processed during feature fusion. In the feature encoding module, the attention mechanism is used to analyze the correlation and dependency between appearance features and motion features. By calculating the correlation scores between features, the system can assign a weight to each feature. Features with strong dependencies have a greater impact on subsequent feature fusion and generation, and are assigned higher weights, while features with weaker dependencies have smaller weights.

[0200] Feature fusion is to combine appearance features and motion features in some way to generate a joint feature representation that can simultaneously express static and dynamic information of the video. The joint feature representation integrates the multi-dimensional information of the video content, which is conducive to a better understanding of the overall semantics of the video. In the feature encoding module, feature fusion is performed using the previously calculated feature dependencies. The fusion process may include weighted summation, splicing and other operations, with the aim of integrating information of different dimensions into a unified feature representation. The final joint feature representation contains both static visual information (such as color, objects) and dynamic motion information (such as object movement, scene changes).

[0201] Although the joint feature representation contains rich information, it may have a high dimension. Therefore, it is necessary to compress and fuse the features through a multi-layer perceptron (MLP) or linear mapping to reduce redundant information and generate the final video features. The generated joint feature representation is input into the multi-layer perceptron for processing. The multi-layer perceptron maps the features to a lower dimension while retaining key information through a series of linear transformations and nonlinear activation functions. Through this process, the final video features generated can effectively express the core content of the video while reducing the dimension of the feature space.

[0202] This embodiment can effectively fuse static information (such as color, scene) and dynamic information (such as object motion trajectory) of the video through the feature encoding module. This fusion method can accurately capture the complex semantics of the video content and improve the accuracy and vividness of subtitle generation.

[0203] In one embodiment, the above S40 includes:

[0204] S401, inputting the video features into an emotion feature encoding module;

[0205] S402, the emotion feature encoding module selects a plurality of candidate emotion categories related to the video feature from an emotion category library;

[0206] S403, generating a preliminary emotion category distribution based on the similarity analysis between the video features and the candidate emotion categories, wherein the preliminary emotion category distribution indicates the relevance of each candidate emotion category to the video content and the corresponding preliminary weight;

[0207] S404, based on the emotion category distribution, using the emotion vocabulary library to optimize the emotion expression of the candidate emotion category, and determining the specific emotion vocabulary under each candidate emotion category;

[0208] S405, reanalyzing the weight of each candidate emotion category according to the specific emotion words under each candidate emotion category to generate a final emotion category distribution;

[0209] S406: Generate emotional features of the video based on the final emotional category distribution.

[0210] In this embodiment, video features are generated by encoding appearance features and motion features to represent the overall content of the video. This step inputs these video features into the emotional feature encoding module for sentiment analysis. Video features (such as objects, scenes, actions, etc.) extracted by convolutional neural networks are passed as input to the emotional feature encoding module. This module is designed to further analyze the emotional information in these features and associate them with predefined emotion categories.

[0211] The emotion category library is a set of predefined emotion labels, such as "happy", "sad", "surprised", etc. The emotion feature encoding module selects emotion categories related to the video content based on the input video features. The emotion feature encoding module preliminarily selects several emotion categories related to the current video features through similarity matching or pre-training models. These emotion categories may be screened through statistical models or deep learning models to ensure the semantic relevance of the emotion categories and the video.

[0212] By calculating the similarity between the video features and each candidate emotion category, a preliminary emotion category distribution is generated, indicating the relevance and weight of each emotion category to the video content. The emotion feature encoding module calculates the similarity score between the video features and each candidate emotion category. Through the similarity score, it can be determined which emotion categories are most likely to reflect the emotional expression in the video. The similarity score is used to generate a preliminary emotion category distribution, marking the preliminary weight of each emotion category.

[0213] The emotional vocabulary library can further refine the expression of emotional categories and make them more expressive. The emotional vocabulary library contains specific emotional words, which are associated with emotional categories and are used to optimize the expression of categories. The emotional feature encoding module combines the emotional category distribution with the emotional vocabulary library to further refine the emotional expression. The emotional vocabulary library contains words such as "anger", "anxiety" or "joy", which can provide more specific semantic labels for emotional categories. The module selects appropriate words based on the emotional category distribution.

[0214] Based on the sentiment vocabulary, the weight of each candidate sentiment category is recalculated to generate the final sentiment category distribution. This distribution is used to further generate sentiment features. After the module further optimizes the sentiment categories using the sentiment vocabulary, the weight of each sentiment category is recalculated to reflect their expression strength of the video sentiment. The final sentiment category distribution more accurately represents the emotional semantics of the video.

[0215] The final generated video emotion feature is based on the optimized emotion category distribution, which can express the specific emotion information of the video and is used for subtitle generation or other emotion analysis tasks. The emotion feature encoding module generates an emotion feature vector based on the final emotion category distribution, which represents the main emotion components in the video. This emotion feature will be used as the input of the subsequent subtitle generation step to ensure that the generated subtitles can correctly convey the emotion information of the video.

[0216] Through the above steps, the emotion feature encoding module of this embodiment can accurately capture the emotion expression in the video, and through the optimization of the emotion category library and the emotion vocabulary library, the emotion feature is made more vivid and accurate. The complex emotion information in the video is effectively expressed, and the generated emotion feature provides a stronger emotion basis for the subsequent subtitle generation.

[0217] In one embodiment, the above S10 includes:

[0218] S101, receiving video data;

[0219] S102, extracting an initial video frame sequence from the video data, wherein the initial video frame sequence is arranged in chronological order;

[0220] S103, performing downsampling processing on the initial video frame sequence to generate a downsampled video frame sequence, wherein the downsampling processing includes temporal downsampling and spatial downsampling.

[0221] In this embodiment, the video data is a data set containing multiple independent video clips. Each video clip contains time-continuous image frames, and the size and format of the video data may be different. The input of video clips is the basis for subsequent data processing. Through the media reading module, the system receives multiple video clips from an input source (such as a local file system or a remote server), uses an adapted video encoding and decoding method (such as H.264, H.265) for format compatibility processing, and stores it as a frame sequence for subsequent processing.

[0222] The system extracts a sequence of image frames for each video clip from the video data, and these frames are arranged in order according to the timestamp. This step ensures that the time order of the video frames is correct to preserve the coherence of the video content. During the frame extraction process, the system reads the timestamp information of each video clip and serializes the video frames according to the timestamp. The frame sequence is arranged in the time order between frames to ensure the logical consistency of the frames during subsequent processing.

[0223] The initial video frame sequence usually contains a large number of frames. Downsampling is used to reduce the number or resolution of frames to reduce the computational burden. This step can be performed in the time domain and the spatial domain. The system performs two types of downsampling operations:

[0224] Temporal downsampling: By reducing the frame rate, representative frames in the video are retained, thereby reducing the number of frames processed. For example, if the video frame rate is 30fps, it can be reduced to 15fps through downsampling.

[0225] Spatial downsampling: Reduce the resolution of each frame (such as from 1920x1080 to 1280x720), reduce the number of pixels in the image, and reduce computational complexity.

[0226] Temporal downsampling is to reduce the number of frames, and spatial downsampling is to reduce the image resolution of each frame. This processing helps to retain the main content of the video when computing and storage resources are limited. The system performs temporal downsampling and spatial downsampling at the set sampling rate to ensure that key video information is maintained while reducing the amount of data. Temporal downsampling reduces unnecessary frames, while spatial downsampling reduces the size of each frame.

[0227] This embodiment can effectively reduce the redundancy and storage space of video frame data through temporal and spatial downsampling processing, improve the efficiency of video processing, and retain key information in the video. This is particularly beneficial for devices with limited resources or real-time processing application scenarios.

[0228] In one embodiment, a video subtitle generation device is provided, and the video subtitle generation device corresponds to the video subtitle generation method in the above embodiment. Figure 3 , Figure 3 The figure is a functional module diagram of a preferred embodiment of the video subtitle generation device of the present invention. The data sampling module 10, the feature extraction module 20, the feature encoding module 30, the emotional feature encoding module 40, the subtitle generation module 50, the emotional guidance analysis module 60, the subtitle processing and feature extraction module 70, the loss analysis module 80, the model optimization module 90 and the subtitle generation module 100. The detailed description of each functional module is as follows:

[0229] The data sampling module 10 is used to acquire video data, perform downsampling processing on the video data, and obtain a video frame sequence after the downsampling processing;

[0230] A feature extraction module 20, configured to extract appearance features from the video frame sequence using a two-dimensional convolutional neural network, and to extract motion features from the video frame sequence using a three-dimensional convolutional neural network;

[0231] A feature encoding module 30, configured to encode the appearance feature and the motion feature through a feature encoding module to generate a video feature;

[0232] The emotion feature encoding module 40 is used to perform emotion encoding on the video features using the emotion feature encoding module based on the emotion category library and the emotion vocabulary library to generate emotion features;

[0233] The subtitle generation module 50 is used to generate an initial subtitle segment and corresponding subtitle features using a predefined start symbol based on a preset initial emotion guidance strength and in combination with the emotion features and video features of the first video frame when processing the first video frame;

[0234] The emotion guidance analysis module 60 is used to analyze the emotion guidance strength of the current video frame by using an attention mechanism based on the video features and emotion features of the current video frame and the subtitle features corresponding to the previous video frame when processing the video frames after the first video frame;

[0235] The subtitle processing and feature extraction module 70 is used to generate a subtitle segment of the current video frame and corresponding subtitle features based on the emotion guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame;

[0236] A loss analysis module 80, used to define a weighted sum of a sentiment center cross entropy loss and a hierarchical sentiment classification loss as a total loss function, and analyze a loss value between a subtitle feature of each video frame and a target sentiment feature corresponding to each video frame according to the total loss function;

[0237] A model optimization module 90, configured to optimize a complete subtitle generation model based on the loss value, wherein the complete subtitle generation model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, and a subtitle feature extraction module;

[0238] The subtitle generation module 100 is used to generate target subtitles based on the optimized complete subtitle generation model.

[0239] In one embodiment, the sentiment guidance analysis module 60 is specifically used to:

[0240] Project the video features of the current video frame and the subtitle features corresponding to the previous video frame into the same embedding space through linear transformation;

[0241] Perform weighted pooling operation on the projected video features and subtitle features to generate extended features;

[0242] Analyzing a cross-modal correlation score between the extended feature and the sentiment feature;

[0243] Based on the cross-modal relevance score, aligning the extended feature with the sentiment feature to form an aligned joint feature;

[0244] dividing the joint feature into a plurality of subspaces;

[0245] Use the attention mechanism to capture the semantic information of each subspace and assign weights to each subspace;

[0246] Based on the weights, the emotion guidance strength of the current video frame is analyzed according to the semantic information of the weighted subspace.

[0247] In one embodiment, the subtitle processing and feature extraction module 70 is specifically used for:

[0248] The emotional guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame are input into the long short-term memory neural network based on the attention mechanism;

[0249] Perform weighted processing on the input emotional guidance strength, the subtitle segment generated by the previous video frame, and the corresponding subtitle features;

[0250] Based on the weighted emotional guidance strength, subtitle segments and corresponding subtitle features, the subtitle segments and corresponding subtitle features of the current video frame are generated through a long short-term memory neural network.

[0251] In one embodiment, the feature extraction module 20 is specifically used for:

[0252] Performing normalization or standardization on the video frame sequence;

[0253] The processed video frame sequence is input into the two-dimensional convolutional neural network frame by frame, and convolution and pooling operations are performed layer by layer on each frame image to extract static appearance features from each frame image;

[0254] The processed video frame sequence is input into a three-dimensional convolutional neural network to perform three-dimensional convolution and pooling operations in spatial and temporal dimensions to extract cross-frame motion features from the video frame sequence, wherein the motion features include dynamic change information and object motion trajectory in the video.

[0255] In one embodiment, the feature encoding module 30 is specifically used to:

[0256] Inputting the appearance features and motion features into a feature encoding module;

[0257] The feature encoding module uses an encoding method based on an attention mechanism to capture the dependency between appearance features and motion features;

[0258] Based on the dependency relationship, the appearance feature and the motion feature are fused by a feature encoding module to generate a joint feature representation, where the joint feature representation includes static information and dynamic information of the video;

[0259] The joint features are compressed and fused through a multi-layer perceptron or linear mapping to generate the final video features.

[0260] In one embodiment, the emotion feature encoding module 40 is specifically used to:

[0261] Inputting the video features into an emotion feature encoding module;

[0262] The emotion feature encoding module selects a plurality of candidate emotion categories related to the video feature from an emotion category library;

[0263] Based on the similarity analysis between the video features and the candidate emotion categories, a preliminary emotion category distribution is generated, wherein the preliminary emotion category distribution indicates the relevance of each candidate emotion category to the video content and the corresponding preliminary weight;

[0264] Based on the emotion category distribution, using the emotion vocabulary library to optimize the emotion expression of the candidate emotion category, and determining the specific emotion vocabulary under each candidate emotion category;

[0265] According to the specific emotional words under each candidate emotional category, the weight of each candidate emotional category is re-analyzed to generate the final emotional category distribution;

[0266] Based on the final emotion category distribution, an emotion feature of the video is generated.

[0267] In one embodiment, the data sampling module 10 is specifically used for:

[0268] receiving video data;

[0269] Extracting an initial video frame sequence from the video data, wherein the initial video frame sequence is arranged in chronological order;

[0270] Downsampling is performed on the initial video frame sequence to generate a downsampled video frame sequence, wherein the downsampling includes temporal downsampling and spatial downsampling.

[0271] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a video subtitle generation method are implemented.

[0272] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user side of a video subtitle generation method

[0273] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0274] Acquire video data, perform downsampling processing on the video data, and obtain a video frame sequence after the downsampling processing;

[0275] Extracting appearance features from the video frame sequence using a two-dimensional convolutional neural network, and extracting motion features from the video frame sequence using a three-dimensional convolutional neural network;

[0276] Encoding the appearance feature and the motion feature through a feature encoding module to generate a video feature;

[0277] Based on the emotion category library and the emotion vocabulary library, using the emotion feature encoding module to perform emotion encoding on the video features to generate emotion features;

[0278] When processing the first video frame, based on the preset initial emotion guidance strength, combined with the emotion feature and video feature of the first video frame, a predefined starting symbol is used to generate an initial subtitle segment and a corresponding subtitle feature;

[0279] When processing the video frames after the first video frame, the attention mechanism is used to analyze the emotional guidance strength of the current video frame based on the video features, emotional features of the current video frame and the subtitle features corresponding to the previous video frame;

[0280] Generate a subtitle segment of the current video frame and corresponding subtitle features based on the emotion guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame;

[0281] A weighted sum of a sentiment center cross entropy loss and a hierarchical sentiment classification loss is defined as a total loss function, and a loss value between a caption feature of each video frame and a target sentiment feature corresponding to each video frame is analyzed according to the total loss function;

[0282] Optimizing a complete subtitle generation model based on the loss value, wherein the complete subtitle generation model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, and a subtitle feature extraction module;

[0283] Generate target subtitles based on the optimized complete subtitle generation model.

[0284] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0285] Acquire video data, perform downsampling processing on the video data, and obtain a video frame sequence after the downsampling processing;

[0286] Extracting appearance features from the video frame sequence using a two-dimensional convolutional neural network, and extracting motion features from the video frame sequence using a three-dimensional convolutional neural network;

[0287] Encoding the appearance feature and the motion feature through a feature encoding module to generate a video feature;

[0288] Based on the emotion category library and the emotion vocabulary library, using the emotion feature encoding module to perform emotion encoding on the video features to generate emotion features;

[0289] When processing the first video frame, based on the preset initial emotion guidance strength, combined with the emotion feature and video feature of the first video frame, a predefined starting symbol is used to generate an initial subtitle segment and a corresponding subtitle feature;

[0290] When processing the video frames after the first video frame, the attention mechanism is used to analyze the emotional guidance strength of the current video frame based on the video features, emotional features of the current video frame and the subtitle features corresponding to the previous video frame;

[0291] Generate a subtitle segment of the current video frame and corresponding subtitle features based on the emotion guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame;

[0292] A weighted sum of a sentiment center cross entropy loss and a hierarchical sentiment classification loss is defined as a total loss function, and a loss value between a caption feature of each video frame and a target sentiment feature corresponding to each video frame is analyzed according to the total loss function;

[0293] Optimizing a complete subtitle generation model based on the loss value, wherein the complete subtitle generation model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, and a subtitle feature extraction module;

[0294] Generate target subtitles based on the optimized complete subtitle generation model.

[0295] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0296] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0297] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0298] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for generating video subtitles, characterized in that: The following steps are involved: Acquire video data, perform downsampling processing on the video data, and obtain a video frame sequence after the downsampling processing; Extracting appearance features from the video frame sequence using a two-dimensional convolutional neural network, and extracting motion features from the video frame sequence using a three-dimensional convolutional neural network; Encoding the appearance feature and the motion feature through a feature encoding module to generate a video feature; Based on the emotion category library and the emotion vocabulary library, using the emotion feature encoding module to perform emotion encoding on the video features to generate emotion features; When processing the first video frame, based on the preset initial emotion guidance strength, combined with the emotion feature and video feature of the first video frame, a predefined starting symbol is used to generate an initial subtitle segment and a corresponding subtitle feature; When processing the video frames after the first video frame, the attention mechanism is used to analyze the emotional guidance strength of the current video frame based on the video features, emotional features of the current video frame and the subtitle features corresponding to the previous video frame; Generate a subtitle segment of the current video frame and corresponding subtitle features based on the emotion guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame; A weighted sum of a sentiment center cross entropy loss and a hierarchical sentiment classification loss is defined as a total loss function, and a loss value between a caption feature of each video frame and a target sentiment feature corresponding to each video frame is analyzed according to the total loss function; Optimizing a complete subtitle generation model based on the loss value, wherein the complete subtitle generation model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, and a subtitle feature extraction module; Generate target subtitles based on the optimized complete subtitle generation model.

2. The video subtitle generation method according to claim 1, characterized in that: Based on the video features and emotional features of the current video frame and the subtitle features corresponding to the previous video frame, the attention mechanism is used to analyze the emotional guidance strength of the current video frame, including: Project the video features of the current video frame and the subtitle features corresponding to the previous video frame into the same embedding space through linear transformation; Perform weighted pooling operation on the projected video features and subtitle features to generate extended features; Analyzing a cross-modal correlation score between the extended feature and the sentiment feature; Based on the cross-modal relevance score, aligning the extended feature with the sentiment feature to form an aligned joint feature; dividing the joint feature into a plurality of subspaces; Use the attention mechanism to capture the semantic information of each subspace and assign weights to each subspace; Based on the weights, the emotion guidance strength of the current video frame is analyzed according to the semantic information of the weighted subspace.

3. The video subtitle generation method according to claim 1, characterized in that: Based on the emotional guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame, the subtitle segment and corresponding subtitle features of the current video frame are generated, including: The emotional guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame are input into the long short-term memory neural network based on the attention mechanism; Perform weighted processing on the input emotional guidance strength, the subtitle segment generated by the previous video frame, and the corresponding subtitle features; Based on the weighted emotional guidance strength, subtitle segments and corresponding subtitle features, the subtitle segments and corresponding subtitle features of the current video frame are generated through a long short-term memory neural network.

4. The video subtitle generation method according to claim 1, characterized in that: Extracting appearance features from the video frame sequence using a two-dimensional convolutional neural network, and extracting motion features from the video frame sequence using a three-dimensional convolutional neural network, including: Performing normalization or standardization on the video frame sequence; The processed video frame sequence is input into the two-dimensional convolutional neural network frame by frame, and convolution and pooling operations are performed layer by layer on each frame image to extract static appearance features from each frame image; The processed video frame sequence is input into a three-dimensional convolutional neural network to perform three-dimensional convolution and pooling operations in spatial and temporal dimensions to extract cross-frame motion features from the video frame sequence, wherein the motion features include dynamic change information and object motion trajectory in the video.

5. The video subtitle generation method as claimed in claim 1, characterized in that: The appearance feature and the motion feature are encoded by a feature encoding module to generate a video feature, including: Inputting the appearance features and motion features into a feature encoding module; The feature encoding module uses an encoding method based on an attention mechanism to capture the dependency between appearance features and motion features; Based on the dependency relationship, the appearance feature and the motion feature are fused by a feature encoding module to generate a joint feature representation, where the joint feature representation includes static information and dynamic information of the video; The joint features are compressed and fused through a multi-layer perceptron or linear mapping to generate the final video features.

6. The video subtitle generation method according to claim 1, characterized in that: Based on the emotion category library and the emotion vocabulary library, the emotion feature encoding module is used to perform emotion encoding on the video feature to generate the emotion feature, including: Inputting the video features into an emotion feature encoding module; The emotion feature encoding module selects a plurality of candidate emotion categories related to the video feature from an emotion category library; Based on the similarity analysis between the video features and the candidate emotion categories, a preliminary emotion category distribution is generated, wherein the preliminary emotion category distribution indicates the relevance of each candidate emotion category to the video content and the corresponding preliminary weight; Based on the emotion category distribution, using the emotion vocabulary library to optimize the emotion expression of the candidate emotion category, and determining the specific emotion vocabulary under each candidate emotion category; According to the specific emotional words under each candidate emotional category, the weight of each candidate emotional category is re-analyzed to generate the final emotional category distribution; Based on the final emotion category distribution, an emotion feature of the video is generated.

7. The video subtitle generation method according to claim 1, characterized in that: Acquiring video data, performing downsampling processing on the video data, and obtaining a video frame sequence after the downsampling processing, including: receiving video data; Extracting an initial video frame sequence from the video data, wherein the initial video frame sequence is arranged in chronological order; Downsampling is performed on the initial video frame sequence to generate a downsampled video frame sequence, wherein the downsampling includes temporal downsampling and spatial downsampling.

8. A video subtitle generation device, characterized in that: The video subtitle generating device comprises: A data sampling module is used to acquire video data, perform downsampling processing on the video data, and obtain a video frame sequence after the downsampling processing; A feature extraction module, used to extract appearance features from the video frame sequence using a two-dimensional convolutional neural network, and to extract motion features from the video frame sequence using a three-dimensional convolutional neural network; A feature encoding module, used for encoding the appearance feature and the motion feature through the feature encoding module to generate a video feature; An emotion feature encoding module is used to perform emotion encoding on the video features based on an emotion category library and an emotion vocabulary library using the emotion feature encoding module to generate emotion features; A subtitle generation module is used to generate an initial subtitle segment and corresponding subtitle features using a predefined starting symbol based on a preset initial emotion guidance strength, combined with the emotion features and video features of the first video frame when processing the first video frame; The emotion guidance analysis module is used to analyze the emotion guidance strength of the current video frame using an attention mechanism based on the video features and emotion features of the current video frame and the subtitle features corresponding to the previous video frame when processing the video frames after the first video frame; A subtitle processing and feature extraction module, for generating a subtitle segment of a current video frame and corresponding subtitle features based on the emotional guidance strength of the current video frame and the subtitle segment and subtitle features generated by the previous video frame; A loss analysis module, used to define a weighted sum of a sentiment center cross entropy loss and a hierarchical sentiment classification loss as a total loss function, and analyze a loss value between a subtitle feature of each video frame and a target sentiment feature corresponding to each video frame according to the total loss function; A model optimization module, used for optimizing a complete subtitle generation model based on the loss value, wherein the complete subtitle generation model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, a feature encoding module, an emotional feature encoding module, and a subtitle feature extraction module; The subtitle generation module is used to generate target subtitles based on the optimized complete subtitle generation model.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a video subtitle generation program stored in the memory and executable on the processor. When the video subtitle generation program is executed by the processor, the steps of the video subtitle generation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a video subtitle generation program, and when the video subtitle generation program is executed by the processor, the steps of the video subtitle generation method according to any one of claims 1 to 7 are implemented.

Images