Video comment emotion recognition method, electronic equipment and storage medium
By extracting video and text features at multiple scales and combining them with attention calculations, the problem of semantic misalignment between videos and comments in video comment sentiment recognition is solved, thereby improving the accuracy of sentiment recognition and semantic expressiveness.
Patent Information
- Application Number
- CN202511081917.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-04
- Publication Date
- 2025-11-11
AI Technical Summary
The accuracy of sentiment recognition in existing video comments is poor, mainly due to the semantic misalignment between the video and the comment.
By extracting video and text features at multiple scales from the self-identifying video, extracting consensus features based on the video and text features at multiple scales, and performing attention calculation, video information at multiple temporal granularities is fused into the text to solve the problem of video-text modality misalignment.
It improves the accuracy of sentiment recognition in video comments and enhances the model's ability to understand video features at different time granularities and express the semantics of comment text.
Smart Images
Figure CN120932218A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a video comment emotion recognition method, electronic device, and storage medium. Background Technology
[0002] With the development of internet self-media, a large number of videos have emerged on social media platforms, potentially attracting numerous comments. To maintain the security and stability of these platforms, it is necessary to identify the emotions expressed in video comments. However, different comments under the same video may focus on different aspects of the video, indicating a semantic misalignment between the video and the comments, leading to poor accuracy in video comment sentiment recognition tasks.
[0003] Improving the accuracy of emotion recognition in video comments is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention
[0004] This invention provides a video comment emotion recognition method, electronic device, and storage medium to at least solve the problem of poor accuracy in video comment emotion recognition in related technologies.
[0005] This invention provides a video comment sentiment recognition method, comprising: Obtain the video to be identified and its corresponding comments to be identified; Video features at multiple scales are extracted from the video to be identified, and text features are extracted from the comments corresponding to the video to be identified. Consensus features at multiple scales are obtained by extracting video and text features at multiple scales; Attention is calculated based on consensus features at multiple scales to obtain the first fusion feature; The first comment sentiment recognition result is obtained by performing classification calculations based on the first fusion feature and the text feature.
[0006] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described video comment emotion recognition methods.
[0007] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described video comment emotion recognition methods.
[0008] This invention extracts video features at multiple scales from the video to be identified and text features from the corresponding comments to be identified. Based on these multi-scale video and text features, consensus features at multiple scales are extracted. This addresses the problem of video-text modality misalignment by extracting consensus features instead of directly aligning video-text modalities, while simultaneously improving the model's understanding of video features at different temporal granularities. Attention calculation is performed based on these multi-scale consensus features to obtain a first fusion feature, thereby integrating video information at multiple temporal granularities into the text. This enhances the semantic expression of video-related comment text. Classification calculation is then performed based on the first fusion feature to obtain a first comment sentiment recognition result. This addresses the problem of semantic misalignment in video-text caused by different comments' attention positions and angles towards the video, thus improving the accuracy of video comment sentiment recognition. Attached Figure Description
[0009] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 A flowchart illustrating a video comment emotion recognition method provided in an embodiment of the present invention; Figure 2 An architecture diagram of a video comment emotion recognition system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a video comment emotion recognition model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a consensus positioning module provided in an embodiment of the present invention. Detailed Implementation
[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.
[0012] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0013] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0014] Here, we will first explain some key terms used in the embodiments of the present invention.
[0015] Video comment sentiment recognition is a cross-modal video task, meaning it requires consideration of both the comment text and the corresponding video content. In reality, videos and their comments are often semantically misaligned. This misalignment manifests in several ways: different comments focus on different parts of the video; for example, a video might contain multiple events, but one comment might only focus on one event or action, while another comment might evaluate the entire video. Furthermore, different comments offer different perspectives on the video. Even regarding the same event in a video, different users, from their subjective viewpoints, will convey different opinions and emotions. Moreover, comments may not objectively describe what happened in the video, but merely convey opinions and emotions.
[0016] Therefore, if a traditional cross-modal sentiment recognition model is used to perform video comment sentiment recognition tasks, cross-modal learning of video and text modalities in the form of semantic similarity will result in a large loss of video and text information, which in turn leads to poor recognition accuracy.
[0017] To address the issue of semantic misalignment between video and text caused by different comments focusing on different positions and angles of attention to the video, and to improve the accuracy of video comment sentiment recognition, this invention provides a video comment sentiment recognition scheme. This scheme extracts video features at multiple scales from the video to be identified and text features from the corresponding comments. Based on these multi-scale video and text features, consensus features at multiple scales are extracted. By extracting consensus features instead of directly aligning the video text modalities, the scheme solves the problem of video text modal misalignment and improves the model's understanding of video features at different temporal granularities. Attention is calculated based on the consensus features at multiple scales to obtain a first fusion feature, thereby integrating video information at multiple temporal granularities into the text and improving the semantic expressiveness of video-related comment text. Classification is then performed based on the first fusion feature to obtain the first comment sentiment recognition result. This approach effectively solves the problem of semantic misalignment between video and text caused by different comments focusing on different positions and angles of attention to the video, thus improving the accuracy of video comment sentiment recognition.
[0018] The present invention provides a video comment sentiment recognition method. The method is described in detail below, in conjunction with the execution flow of the video comment sentiment recognition method.
[0019] Figure 1 A flowchart illustrating a video comment emotion recognition method provided in an embodiment of the present invention; Figure 2 An architecture diagram of a video comment emotion recognition system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a video comment emotion recognition model provided in an embodiment of the present invention.
[0020] like Figure 1 As shown, the video comment sentiment recognition method provided in this embodiment of the invention may include: S101: obtaining the video to be recognized and its corresponding comment to be recognized.
[0021] S102: Extract video features at multiple scales from the video to be identified, and extract text features from the comments corresponding to the video to be identified.
[0022] S103: Consensus features at multiple scales are obtained by extracting video and text features at multiple scales.
[0023] S104: Attention is calculated based on consensus features at multiple scales to obtain the first fusion feature.
[0024] S105: Classification calculation is performed based on the first fusion feature and text features to obtain the first comment sentiment recognition result.
[0025] The video comment sentiment recognition method provided in this invention can be applied to the training phase or the inference phase of a video comment sentiment recognition model. Therefore, the video comment sentiment recognition method provided in this invention can be applied to, for example... Figure 2 The video comment sentiment recognition system shown includes a model training system and a video comment sentiment recognition device.
[0026] like Figure 2 As shown, the model training system can be supported by multiple AI servers (AI servers 1~n) providing computing power and storage. It utilizes digital human action samples to train a video comment emotion recognition model based on the model training system. In some optional embodiments of this invention, a computing resource pool can be constructed based on the model training system. This pool includes multiple computing threads, allowing the training task of the video comment emotion recognition model to be divided into multiple sub-tasks for parallel execution. Both model parallel training and data parallel training methods can be used. The model training system can be deployed in a cloud computing center. The equipment structure of the cloud computing center can be similar to... Figure 2 The video comment sentiment recognition device shown includes basic components such as an AI processor, storage, input / output devices, a communication bus, and communication interfaces. It comprises an operating system, database, middleware, and other software environments and application software. The model training system receives manually labeled digital human action samples and can use a stored pre-trained model for fine-tuning to obtain a video comment sentiment recognition model, or it can train a video comment sentiment recognition model using a randomly initialized model framework.
[0027] Video comment emotion recognition devices can be computing devices or terminal devices deployed in cloud computing centers, and their structure is as follows: Figure 2 The video comment sentiment recognition device is shown in the diagram. If a cloud computing center is used, after deploying the trained video comment sentiment recognition model, the terminal device uploads the input video to be detected to the cloud computing center. The trained video comment sentiment recognition model is then used to calculate the sentiment of the input video comment and its corresponding comment, outputting the first sentiment recognition result. If the video comment sentiment recognition device uses a terminal device, the model training system trains the video comment sentiment recognition model and sends it to the terminal device via a communication unit. The model is then deployed in the terminal device's storage. The terminal device receives the input video and performs the video comment sentiment recognition task using its local video comment sentiment recognition model.
[0028] Because different comments focus on different parts of the video, and their evaluation perspectives may also differ, performing video comment sentiment recognition requires, on the one hand, locating the video segment the comment focuses on, and on the other hand, modeling the comment's focus on the video at multiple time scales and from local to global levels. Therefore, this invention provides a structure for a video comment sentiment recognition model as follows: Figure 3 As shown, the video to be identified and its corresponding comment are input into the video comment sentiment recognition model, and the output is the video comment sentiment recognition result. During the model training phase, the video to be identified and its corresponding text are respectively video samples and their corresponding comment samples. During the model inference phase, the video to be identified and its corresponding text are respectively the video to be identified and its corresponding comment. The video comment sentiment recognition model includes a multi-scale video feature extraction module, a consensus localization module, and a multi-scale fusion module. The multi-scale video feature extraction module models video representations from multiple granular time scales. Between representations of different granularities, the consensus localization inference model is used to construct video-comment consensus, thereby solving the problem of semantic misalignment in video comments. Finally, the multi-scale fusion module is used to fuse the multi-scale consensus features, concatenate the text features, and input them into the classifier.
[0029] In this embodiment of the invention, the video comment emotion recognition result can include two categories: opinion recognition result and emotion recognition result, corresponding to... Figure 3 The video comment sentiment recognition model shown includes an opinion classifier and an emotion classifier. Opinion recognition results represent the commenter's attitude towards the video content; the labels for opinion recognition results can include positive, negative, and neutral. Emotion recognition results represent the emotions contained in the comment, which are the emotional reactions and tendencies evoked by watching the video and expressed through the comment. These can include eight labels: trust, joy, surprise, anticipation, disgust, sadness, anger, and fear. Additionally, other opinion or emotion labels can be set.
[0030] Using the model architecture provided in this embodiment of the invention, during the model training phase, video samples and their corresponding comment samples are acquired, along with annotation information (ground values of comment sentiment recognition results). The video comment sentiment recognition model before training is denoted as the recognition model. The video samples and their corresponding comment samples are input into the recognition model, and the output comment sentiment recognition result is denoted as the second comment sentiment recognition result. The second comment sentiment recognition result and the corresponding annotation information are used to calculate the loss value. The model parameters are updated using the loss value until the iteration training termination condition is met, thus obtaining the video comment sentiment recognition model. The iteration training termination condition can be that the number of iterations of the recognition model reaches a preset number of iterations, or that the loss value of the recognition model is less than a preset loss value.
[0031] During the model inference stage, the video to be identified and the comment to be identified are input into the video comment sentiment recognition model, and the output comment sentiment recognition result is recorded as the first comment sentiment recognition result.
[0032] The video comment sentiment recognition method provided in this invention extracts video features at multiple scales from the video to be identified and text features from the corresponding comments to be identified. Based on these video and text features, consensus features at multiple scales are extracted. This approach addresses the problem of video-text modality misalignment by extracting consensus features instead of directly aligning video and text modalities, while simultaneously improving the model's understanding of video features at different temporal granularities. Attention calculation is performed based on the consensus features at multiple scales to obtain a first fusion feature, thereby fusing video information at multiple temporal granularities into the text. This enhances the semantic expressiveness of video-related comment text. Classification calculation is then performed based on the first fusion feature to obtain a first comment sentiment recognition result. This method can solve the problem of video-text semantic misalignment caused by different comments' attention positions and angles towards the video, thus improving the accuracy of video comment sentiment recognition.
[0033] Based on the above embodiments, the present invention will further describe the model structure of the video comment emotion recognition model.
[0034] like Figure 3 As shown, in the video comment sentiment recognition model provided in this embodiment of the invention, a visual encoder is used to extract initial video features from the input video, and a text encoder is used to extract text features from the comment text.
[0035] The visual encoder can employ an I3D video pre-trained model. For the input video, all its frames can be extracted, and the I3D video pre-trained model can be used for video feature extraction. The extraction unit can be set to 16 frames, with a stride of 8 frames. That is, every 16 frames is considered a clip, yielding a feature dimension of length 1024. The model slides along the time axis with a stride of 8 frames, with a maximum length of 180 clips. Excess clips are truncated, and insufficient clips are padded with zeros to obtain the initial video features. That is, each clip is extracted as an initial video feature.
[0036] The text encoder can use a RoBERTa pre-trained model. Input the comment text into the text encoder; the maximum length can be set to 512, and the text features will be obtained. .
[0037] like Figure 3As shown, in the video comment sentiment recognition model provided in this embodiment of the invention, the multi-scale video feature extraction module is used to model video information at multiple time scales. Before inputting the initial video features into the multi-scale video feature extraction module, the feature length can be converted from 1024 to 768 through a fully connected layer and the ReLU activation function.
[0038] The multi-scale video feature extraction module includes multiple feature extraction layers. Each subsequent feature extraction layer takes the output of the previous layer as input, and each layer performs further feature extraction based on its input video features. In this embodiment, the feature extraction layers can employ a Convolutional Neural Network (CNN). The receptive field window of each feature extraction layer can be set to 3, and the stride to 1. This means that one video feature is extracted for every 3 clips, and the process continues by sliding one clip forward along the time axis to extract the next set of clips as one video feature, and so on. Thus, through multiple feature extraction layers stacked together, the receptive field of the video is expanded from bottom to top. In this embodiment, the feature dimension of each feature extraction layer can be... (That is, the feature dimension is 768 and the temporal length is 180), and ReLU is used as the activation function between layers. Therefore, the video features output by each feature extraction layer are multi-scale video features. In this embodiment of the invention, the multi-scale video feature extraction module includes four feature extraction layers, outputting four sets of video features at different scales.
[0039] In this embodiment of the invention, a consensus localization module is set for video features at each scale to extract consensus features from the video features and text features at the corresponding scale.
[0040] In this embodiment of the invention, S103, which extracts consensus features based on video features and text features at multiple scales, may include: performing cross-modal attention calculations on the first consensus feature with the video features and the text features at each scale to obtain the consensus feature.
[0041] Figure 4 This is a schematic diagram of the structure of a consensus positioning module provided in an embodiment of the present invention.
[0042] like Figure 4 As shown, the consensus localization module provided in this embodiment of the invention may include a consensus encoder and a temporal cue localization module. The consensus encoder may employ a Transformer encoder to better model the deep semantic relevance between videos and comments.
[0043] The key to identifying video features relevant to comments is to develop a reasonable query to "bridge" the semantic gap between comments and videos. For example... Figure 4 As shown, based on the Transformer encoder structure, the consensus encoder uses a trainable consensus feature (consensus token), denoted as the first consensus feature, which is inserted between the video features and text features as input to the consensus encoder. This embodiment of the invention masks the attentional connection between video features and text features, but they can exchange information through the consensus token in the consensus encoder. Based on the consensus space, the first consensus feature is subjected to cross-modal attention computation with both video features and text features. That is, cross-modal attention computation is performed between video features and the first consensus feature (using the first consensus feature as the query vector and the video features as the key) to establish an attentional connection between video features and the consensus token. Similarly, cross-modal attention computation is performed between text features and the first consensus feature (using the first consensus feature as the query vector and the text features as the key) to establish an attentional connection between text features and the consensus token. Thus, video features and text features can exchange information through the consensus token, resulting in a second consensus feature, which represents the semantic consensus between video features and text features. The consensus token output by the consensus encoder (denoted as the second consensus feature) is used as the consensus feature for querying and locating video information.
[0044] In this way, consensus features can learn video features. and comment text features This design effectively bridges the semantic gap between videos and comments by sharing information between them.
[0045] In the above implementation, the video features input to the consensus positioning module are video features of the corresponding scale, and the input text features are the text features output by the text encoder. That is, the text features input to each consensus positioning module are the same.
[0046] The video features output by the consensus positioning module are fused video features, and the text features output are fused text features.
[0047] To further deepen the correlation between multi-scale features, in some optional embodiments of this invention, for each scale's video features and its corresponding text features, a first consensus feature is performed with the video features and text features respectively to obtain a consensus feature. This can include: for the video features extracted from the video to be identified at the first scale, performing cross-modal attention calculations with the video features at the current scale and the text features extracted from the comment to be identified respectively to obtain the consensus feature at the current scale; for video features at other scales, performing cross-modal calculations with the video features at the current scale and the fused text features output from the cross-modal attention calculation at the previous scale respectively to obtain the consensus feature at the current scale. That is to say, the text features input to the consensus localization module corresponding to the first scale are the text features output by the text encoder, while the text features input to other consensus localization modules are the fused text features output by the previous scale. Thus, the consensus localization module at the current scale can inherit the fused text features from the consensus localization module at the previous scale, deepening the understanding of multi-scale text features.
[0048] In some optional embodiments of the present invention, for video features and their corresponding text features at each scale, the first consensus feature is subjected to cross-modal attention calculation with the video features and text features respectively to obtain consensus features. This may further include: for video features extracted from the video to be identified at the first scale, the first consensus feature is subjected to cross-modal attention calculation with the video features and text features at the current scale respectively to obtain consensus features at the current scale; for video features at other scales, the consensus feature output by the cross-modal attention calculation of the previous scale is used as the first consensus feature of the current scale, and the first consensus feature is subjected to cross-modal attention calculation with the video features and text features at the current scale respectively to obtain consensus features at the current scale. That is to say, the first consensus feature input by the consensus positioning module corresponding to the first scale is the initial first consensus feature, while the first consensus feature input by other consensus positioning modules is the second consensus feature output by the previous scale. Thus, the consensus positioning module at the current scale can inherit the consensus feature output by the consensus positioning module at the previous scale, deepening the understanding of multi-scale consensus features.
[0049] In some optional embodiments of the present invention, for video features and their corresponding text features at each scale, the first consensus feature is subjected to cross-modal attention calculation with the video features and text features respectively to obtain consensus features. This may include: for video features extracted from the first scale of the video to be identified, the first consensus feature is subjected to cross-modal attention calculation with the video features at the current scale and the text features extracted from the comment to be identified respectively to obtain consensus features at the current scale; for video features at other scales, the consensus feature output by the cross-modal attention calculation of the previous scale is used as the first consensus feature at the current scale, and the fused text features output by the cross-modal attention calculation of the previous scale are used as the text features input at the current scale. The first consensus feature is subjected to cross-modal attention calculation with the video features and text features at the current scale respectively to obtain consensus features at the current scale. In other words, the text features input to the consensus localization module corresponding to the first scale are the text features output by the text encoder, and the first consensus feature input to the consensus localization module corresponding to the first scale is the initial first consensus feature. The text features input to other consensus localization modules are the fused text features output by the previous scale, and the first consensus feature input is the second consensus feature output by the previous scale. Thus, the consensus localization module of the current scale can inherit the fused text features of the consensus localization module of the previous scale as well as the consensus features output by the consensus localization module of the previous scale, deepening the model's understanding of multi-scale text features and multi-scale consensus features.
[0050] To further understand video features, it is necessary not only to focus on the relationship between the video and the comments, but also to pay attention to the video's contextual narrative. This is achieved through the temporal clue positioning module in the consensus positioning module provided in this embodiment of the invention.
[0051] In this embodiment of the invention, the first consensus feature is subjected to cross-modal attention calculation with video features and text features respectively to obtain consensus features, including: performing cross-modal attention calculation with video features and text features at the current scale respectively to obtain a second consensus feature at the current scale; calculating the global temporal localization weight of the video temporal segment based on the video features and the second consensus feature at the current scale; and performing a weighted summation of the video features at the current scale based on the global temporal localization weight to obtain the consensus feature at the current scale; wherein, the video temporal segment is a continuous video frame in the video to be identified that satisfies the video feature similarity condition.
[0052] In real-world scenarios, a video may contain multiple events, and the durations of these events may vary. For example, the first event might last for a much longer number of frames than the second event. This can lead to an overemphasis on the first event while neglecting the second. However, in video comment sentiment recognition tasks, it is desirable to treat all events equally. Therefore, in this embodiment of the invention, consecutive video frames pointing to the same event are defined as a video temporal segment, where the similarity condition is that they qualitatively point to the same event.
[0053] In some optional embodiments of the present invention, calculating the global temporal localization weight of a video temporal segment based on the video features at the current scale and the second consensus features may include: using the second consensus features at the current scale as the query vector and the video features at the current scale as the key, performing multi-head attention calculation to obtain a multi-head attention score; inputting the multi-head attention score into a long short-term memory network to obtain a cell state representation; and using a filtering function to model the global temporal localization weight of the video over time on the cell state representation to obtain the global temporal localization weight.
[0054] In other words, the second consensus feature (as the query vector) and the previously encoded video features (as the key) can be input into the multi-head attention network to obtain a multi-head attention score as a temporal feature. The purpose of this step is to utilize As a temporal feature, temporal attention trends in different vector spaces are considered. Then, the temporal features are... The input is fed into a Long Short-Term Memory (LSTM) block to obtain a cell state representation. The cell state representation is then filtered by a global temporal positioning weight applied to the video over time. Modeling is then performed. The resulting set of global temporal localization weights has a relatively smooth distribution, which leads to the problem described above of not being able to equally focus on events in the video. In this embodiment of the invention, the video features at the current scale and the global temporal localization weights are input into a filtering function to obtain discrete attention weights. These discrete attention weights are then compared with the video features at the current scale. Multiply and sum them to obtain the consensus feature obtained in S102, denoted as . This consensus feature can be called a video cue feature, which enables the removal of redundancy from video frames that have similar events.
[0055] In the embodiments of the invention, the filtering function used by the temporal cue localization module can be a ReLU function, or it can be replaced with other filtering functions. Besides using a Long Short-Term Memory network and a filtering function, a Multilayer Perceptron (MLP) can also be used to achieve multi-head attention-based scoring. The final consensus features (video cue features) were obtained.
[0056] Different comments may focus on the feature expressions at different time scales of short videos. In this embodiment of the invention, attention calculation is performed based on consensus features at multiple scales to obtain a first fusion feature, which may include: using text features as the query vector and consensus features at multiple scales as the key values, performing cross-modal attention calculation to obtain the first fusion feature. Figure 3 As shown, by using the multi-scale fusion module provided in this embodiment of the invention, multi-scale video cue features and comment text features can be adaptively selected for fusion, which can more accurately and effectively enhance the semantic representation of opinions and emotions.
[0057] The multi-scale fusion module guides comments by modeling video features at different time scales. In this module, a multi-scale attention network is first used to calculate the attention weights between consensus features (video cue features) and text features at different time scales. Then, the obtained attention weights are used to accumulate the consensus features, resulting in a first fused feature that comprehensively considers different time scales. This operation effectively combines consensus features with text features, thereby better guiding classification tasks.
[0058] Then, the first fusion feature is copied as a text feature. Then, with text features Connecting along the feature dimensions, the concatenated features are denoted as video context-aware comment semantic features. This connection allows video and text information to be merged into a unified representation. For example, if the length of the first fusion feature described in the above embodiment is 1, and the length of the text feature is 512, then the first fusion feature is copied 512 times and then concatenated with the text feature.
[0059] In this embodiment of the invention, both the opinion classifier and the sentiment classifier can be composed of a pooling layer and a softmax function, which are used to output the probability values of opinion labels and sentiment labels, respectively.
[0060] In this embodiment of the invention, when training the video comment sentiment recognition model, the Adam Weight Decay Optimizer (AdamW) can be used for loss optimization calculation, employing a linear learning rate optimization strategy and a warmup strategy with a parameter set to 0.1. The number of training iterations can be set to 120, and the model with the most recent performance can be obtained through early stopping and fine-tuning strategies.
[0061] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0062] Embodiments of the present invention also provide a video comment sentiment recognition device, which may include: an acquisition module for acquiring a video to be recognized and its corresponding comment to be recognized; a feature extraction module for extracting video features at multiple scales from the video to be recognized and extracting text features from the comment corresponding to the video to be recognized; a consensus localization module for extracting consensus features at multiple scales based on the video features and text features at multiple scales; a multi-scale fusion module for performing attention calculation based on the consensus features at multiple scales to obtain a first fusion feature; and a classification calculation module for performing classification calculation based on the first fusion feature and text features to obtain a first comment sentiment recognition result.
[0063] In this embodiment of the invention, the consensus localization module extracts consensus features at multiple scales based on video features and text features at multiple scales. This may include: performing cross-modal attention calculations on the first consensus feature with the video features and the text features at each scale for each video feature and its corresponding text features, respectively, to obtain the consensus feature.
[0064] The consensus localization module performs cross-modal attention calculations on the video features and their corresponding text features at each scale to obtain consensus features. This can include: for the video features extracted from the first scale of the video to be identified, performing cross-modal attention calculations on the first consensus feature with the video features at the current scale and the text features extracted from the comment to be identified to obtain the consensus features at the current scale; for video features at other scales, performing cross-modal calculations on the first consensus feature with the video features at the current scale and the fused text features output from the cross-modal attention calculations at the previous scale to obtain the consensus features at the current scale.
[0065] The consensus localization module performs cross-modal attention calculations on the video features and their corresponding text features at each scale to obtain consensus features. It may also include: for the video features extracted from the video to be identified at the first scale, performing cross-modal attention calculations on the first consensus feature with the video features and text features at the current scale to obtain consensus features at the current scale; for video features at other scales, the consensus feature output from the cross-modal attention calculation of the previous scale is used as the first consensus feature at the current scale, and the first consensus feature is then performed cross-modal attention calculations on the video features and text features at the current scale to obtain consensus features at the current scale.
[0066] The consensus localization module performs cross-modal attention calculations on the first consensus feature with video features and text features respectively to obtain consensus features. It may also include: performing cross-modal attention calculations on the video features and their corresponding text features at the current scale to obtain the second consensus feature at the current scale; calculating the global temporal localization weight of the video temporal segment based on the video features and the second consensus feature at the current scale; and performing a weighted summation of the video features at the current scale based on the global temporal localization weight to obtain the consensus feature at the current scale. Here, the video temporal segment is a continuous video frame in the video to be identified that satisfies the video feature similarity condition.
[0067] In this embodiment of the invention, the consensus localization module calculates the global temporal localization weight of a video time sequence segment based on the video features at the current scale and the second consensus features. This can include: using the second consensus features at the current scale as the query vector and the video features at the current scale as the key, performing multi-head attention calculation to obtain a multi-head attention score; inputting the multi-head attention score into a long short-term memory network to obtain a unit state representation; and using a filtering function to model the global temporal localization weight of the video over time using the unit state representation to obtain the global temporal localization weight.
[0068] In this embodiment of the invention, the multi-scale fusion module performs attention calculation based on consensus features of multiple scales to obtain a first fusion feature, which may include: using text features as query vectors and consensus features of multiple scales as key values to perform cross-modal attention calculation to obtain the first fusion feature.
[0069] For a description of the features in the embodiment corresponding to the video comment emotion recognition device, please refer to the relevant description of the embodiment corresponding to the video comment emotion recognition method, which will not be repeated here.
[0070] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the video comment emotion recognition method.
[0071] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described embodiments of the video comment sentiment recognition method when running.
[0072] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0073] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described video comment emotion recognition method embodiments.
[0074] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the video comment emotion recognition method.
[0075] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0076] The above provides a detailed description of the video comment emotion recognition method, electronic device, and storage medium provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.
Claims
1. A video comment sentiment recognition method, characterized in that, include: Obtain the video to be identified and its corresponding comments to be identified; Video features at multiple scales are extracted from the video to be identified, and text features are extracted from the comments corresponding to the video to be identified. Consensus features at multiple scales are obtained by extracting video and text features at multiple scales; Attention is calculated based on consensus features at multiple scales to obtain the first fusion feature; The first comment sentiment recognition result is obtained by performing classification calculations based on the first fusion feature and the text feature.
2. The video comment emotion recognition method according to claim 1, characterized in that, Consensus features at multiple scales are obtained by extracting video and text features at multiple scales, including: For each scale of video features and its corresponding text features, the first consensus features are subjected to cross-modal attention calculation with the video features and text features respectively to obtain the consensus features.
3. The video comment emotion recognition method according to claim 2, characterized in that, For each scale of video features and its corresponding text features, the first consensus features are subjected to cross-modal attention calculation with the video features and text features respectively to obtain the consensus features, including: For the video features at the first scale extracted from the video to be identified, the first consensus feature is combined with the video features at the current scale and the text features extracted from the comment to be identified to perform cross-modal attention calculation to obtain the consensus feature at the current scale. For video features at other scales, the first consensus feature is combined with the video features at the current scale and the fused text features output by the cross-modal attention calculation at the previous scale to perform cross-modal calculations to obtain the consensus feature at the current scale.
4. The video comment emotion recognition method according to claim 2, characterized in that, For each scale of video features and its corresponding text features, the first consensus features are subjected to cross-modal attention calculation with the video features and text features respectively to obtain the consensus features, including: For the video features at the first scale extracted from the video to be identified, the first consensus features are subjected to cross-modal attention calculation with the video features and text features at the current scale, respectively, to obtain the consensus features at the current scale; For video features at other scales, the consensus features output by the cross-modal attention calculation at the previous scale are used as the first consensus features at the current scale. The first consensus features are then subjected to cross-modal attention calculations with the video features and text features at the current scale to obtain the consensus features at the current scale.
5. The video comment emotion recognition method according to claim 2, characterized in that, For each scale of video features and its corresponding text features, the first consensus features are subjected to cross-modal attention calculation with the video features and text features respectively to obtain the consensus features, including: For the video features at the first scale extracted from the video to be identified, the first consensus feature is combined with the video features at the current scale and the text features extracted from the comment to be identified to perform cross-modal attention calculation to obtain the consensus feature at the current scale. For video features at other scales, the consensus features output by the cross-modal attention calculation at the previous scale are used as the first consensus features at the current scale, and the fused text features output by the cross-modal attention calculation at the previous scale are used as the text features input at the current scale. The first consensus features are then subjected to cross-modal attention calculations with the video features and text features at the current scale to obtain the consensus features at the current scale.
6. The video comment emotion recognition method according to claim 2, characterized in that, The consensus features are obtained by performing cross-modal attention calculations with the first consensus features and video features and text features respectively, including: For the video features and their corresponding text features at the current scale, the first consensus feature is subjected to cross-modal attention calculation with the video features and the text features respectively to obtain the second consensus feature at the current scale; Based on the video features at the current scale and the second consensus features, the global temporal positioning weights of the video temporal segments are calculated. The video features at the current scale are weighted and summed according to the global temporal positioning weights to obtain the consensus features at the current scale. The video time sequence segment is a series of consecutive video frames in the video to be identified that meet the video feature similarity condition.
7. The video comment emotion recognition method according to claim 6, characterized in that, Based on the video features at the current scale and the second consensus features, the global temporal localization weights of the video temporal segments are calculated, including: Using the second consensus feature at the current scale as the query vector and the video feature at the current scale as the key, multi-head attention is calculated to obtain a multi-head attention score. The multi-head attention score is input into the long short-term memory network to obtain the unit state representation; The global temporal positioning weights are obtained by using a filtering function to model the unit state representation with video over time.
8. The video comment emotion recognition method according to claim 6, characterized in that, Attention is calculated based on consensus features at multiple scales to obtain the first fusion feature, including: Using text features as the query vector and consensus features at multiple scales as keys, cross-modal attention calculation is performed to obtain the first fused feature.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the video comment emotion recognition method as described in any one of claims 1 to 8 when executing the computer program.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the video comment emotion recognition method as described in any one of claims 1 to 8.