A visual emotion recognition method, device, equipment and readable storage medium

Abstract

The present application relates to the field of computer vision, and specifically discloses a visual emotion recognition method, device, equipment and readable storage medium, cross-modal consensus information of visual text is constructed through mutual information between visual modalities and text modalities, cross-modal coding of visual text is configured to be encoded based on the cross-modal consensus information of visual text, compared with alignment coding, the expression ability of cross-modal coding of visual text is enhanced, a multi-head attention mechanism of text sentiment evaluation coding, cross-modal coding of visual text and fusion coding is configured in the initial visual model, the weights of each attention head in the model are trained by using sample data set, task target and loss value of each attention head, so that the visual emotion recognition model can more accurately understand the cross-modal sentiment consensus, and then the visual emotion recognition model is called to execute the visual emotion recognition task to be processed, and more accurate visual emotion recognition is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a visual emotion recognition method, apparatus, device, and readable storage medium. Background Technology

[0002] With the development of social networking platforms, online information interaction is no longer limited to text, images, and videos, providing users with rich emotional interaction experiences. This manifests in the fact that images or videos posted on online platforms convey emotional information, and users, while viewing these images or videos, also generate subjective emotional responses, even engaging in emotional interaction with the publishers through comments. Therefore, it is necessary to identify the emotions carried by image or video information to address many related and urgent tasks, such as video image classification, image or video-based question answering, and image or video retrieval and recommendation.

[0003] To learn the sentiment features of images and videos, researchers in related fields have proposed cross-modal fusion learning methods that employ visual and textual modalities to enhance the understanding of images or videos and thus improve the accuracy of sentiment analysis. However, the textual annotations accompanying images or videos are often misaligned with their semantics, which poses a challenge and bottleneck to visual sentiment recognition.

[0004] Improving the visual emotion recognition capabilities of computer vision is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a visual emotion recognition method, apparatus, device, and readable storage medium to improve the visual emotion recognition capability of computer vision, and to enhance the ability to recognize emotions in images and videos.

[0006] To address the aforementioned technical problems, this invention provides a visual emotion recognition method, comprising:

[0007] Obtain the initial visual model and sample dataset;

[0008] Configure the initial visual model with a first attention head based on text sentiment evaluation coding, a second attention head based on visual text cross-modal consensus information, and a third attention head that fuses the text sentiment evaluation coding and the visual text cross-modal coding. After concatenating the loss values ​​of each attention head into the model loss value of the initial visual model, use the sample dataset, the task objective of the visual sentiment recognition task to be processed, and the loss values ​​of each attention head to train the weights of each attention head in the initial visual model, and output the visual sentiment recognition model.

[0009] In response to the visual emotion recognition task to be processed, the visual emotion recognition model is invoked to perform emotion recognition processing on the input data to be recognized, and the visual emotion recognition result is obtained.

[0010] Specifically, according to the task objective of the visual emotion recognition task to be processed, the sample dataset includes image data with annotated text or video data with annotated text; the visual-text cross-modal consensus information is constructed through mutual information between the visual modality and the text modality.

[0011] On the one hand, the visual emotion recognition task to be processed is a video emotion recognition task;

[0012] The second attention head includes a fourth attention head based on video frame text cross-modal consensus information and a fifth attention head based on video text cross-modal consensus information.

[0013] On the other hand, the model loss value is expressed by the following formula:

[0014] ;

[0015] in, The model parameters for the i-th attention head are... The weight of the i-th attention head, Let i be the loss value of the attention head. Let i be the model loss value; i = 0, 1, 2, 3;

[0016] The attention head includes the first attention head, the fourth attention head, the fifth attention head, and the third attention head.

[0017] On the other hand, the loss value of the attention head is expressed by the following formula:

[0018] ;

[0019] The output of the visual emotion recognition model is expressed by the following formula:

[0020] ;

[0021] in, , , , ;

[0022] in, For sentiment classification loss weights, Weighting loss for opinion categories For sentiment classification loss value, Loss value for opinion classification, Here, L is the loss weight calculation function, and L is the concatenated value of the sentiment classification loss value and the opinion classification loss value. It is a fully connected layer. For activation function, For regression function, For the sentiment classification results, For the opinion classification results, The video emotion recognition model outputs the model calculation results based on the input video data and text data.

[0023] On the other hand, the visual emotion recognition task to be processed is a video emotion recognition task;

[0024] The process of training the weights of each attention head in the initial visual model using the sample dataset, the task objective of the visual emotion recognition task to be processed, and the loss values ​​of each attention head, and outputting the visual emotion recognition model, includes:

[0025] For video samples in the sample dataset, multiple video frames are extracted from the video samples, the frame visual features of the video frames are obtained, and time dimension information is added to the frame visual features according to the order of the video frames in the video samples.

[0026] Extract text features based on the annotation text corresponding to the video sample;

[0027] After encoding the frame visual features and the text features with added time dimension information, the video features of the video sample are obtained;

[0028] The video features are input into the initial visual model, and the weights of each attention head in the initial visual model are adjusted according to the loss value of each attention head, and a visual emotion recognition model is output.

[0029] On the other hand, the step of extracting text features based on the annotation text corresponding to the video sample includes:

[0030] Extract the subject and object from the annotation text corresponding to the video sample;

[0031] The text features are obtained by concatenating the subject and object with the search keywords used to obtain the video samples.

[0032] On the other hand, the step of extracting text features based on the annotation text corresponding to the video sample includes:

[0033] If the video sample corresponds to video segmentation information and segmentation event description, then the video sample is divided into multiple sub-video samples according to the video segmentation information, and the corresponding segmentation event description is used as the annotation text of the sub-video sample;

[0034] The video action recognition model is invoked to identify and predict the action of the sub-video sample.

[0035] After extracting video description keywords from the annotation text of the sub-video sample, the action prediction result of the sub-video sample is concatenated with the video description keywords of the sub-video sample to obtain the text features of the sub-video sample;

[0036] The text features of each of the sub-video samples are integrated into the text features of the video sample.

[0037] On the other hand, the step of extracting text features based on the annotation text corresponding to the video sample includes:

[0038] If the video sample corresponds to video segmentation information and segmentation action description, then the video sample is divided into multiple sub-video samples according to the video segmentation information;

[0039] The video description model is invoked to generate description text for each of the sub-video samples;

[0040] After extracting action keywords from the segmented action description and video description keywords from the description text, the action keywords and video description keywords of the sub-video sample are concatenated to obtain the text features of the sub-video sample.

[0041] The text features of each of the sub-video samples are integrated into the text features of the video sample.

[0042] On the other hand, the initial visual model is obtained through the following steps:

[0043] The initial visual model is obtained by constructing a loss function based on visual text matching task, masked text prediction task and consensus information filtering task;

[0044] The visual-text matching task is used to identify the matching status between visual samples and text samples; the masked text prediction task is used to identify the information to be masked based on the context information that is not masked; and the consensus information filtering task is used to filter the consensus information of visual modalities and text modalities from the prior dictionary.

[0045] On the other hand, the loss function corresponding to the visual text matching task is:

[0046] ;

[0047] in, Let S be the loss value for the visual-text matching task, and S be the number of matching pairs between the visual modality and the text modality. For sign functions, when the visual modality matches the text modality A value of 1 indicates a mismatch between the visual and textual modalities. =0, For the i-th visual modality, For the i-th text modality, The probability of matching visual modalities with text modalities.

[0048] On the other hand, the loss function corresponding to the masked text prediction task is:

[0049] ;

[0050] ;

[0051] in, Let S be the first mask text prediction loss value, S be the number of matching pairs between the visual modality and the text modality, and V be the dictionary size of the text modality. Let be the sign function, which is defined when the visual label predicted based on the v-th visual modality matches the masked visual label in the i-th text modality. The value is 1 when the visual label predicted based on the v-th visual modality does not match the masked visual label in the i-th text modality. =0, For the i-th visual modality, For the i-th text modality, To predict the probability of a masked visual label based on the input visual and textual modalities;

[0052] The loss value for predicting the second masked text. Let be the sign function, which is defined when the annotation text predicted based on the v-th visual modality matches the masked annotation text in the i-th text modality. The value is 1 when the annotation text predicted based on the v-th visual modality does not match the masked annotation text in the i-th text modality. =0, Let i be the visual label for the i-th visual modality. Let be the masked annotation text in the i-th text modality. To predict the probability of a masked text modality based on the input visual modality, visual label, and remaining text modality.

[0053] On the other hand, the loss function corresponding to the consensus information filtering task is:

[0054] ;

[0055] in, The loss function for the consensus information filtering task is defined, where K is the amount of prior knowledge related to the target. For vectorized functions, For weight parameters, For the i-th visual feature, For the j-th text feature, To associate relevant information between the i-th visual modality and the j-th text modality, Let be the dictionary value of conditional mutual information between the i-th visual modality, the j-th text modality, and the relevant information relating the i-th visual modality and the j-th text modality. For information between visual and textual modalities, Let be the conditional mutual information dictionary values ​​for the i-th visual modality, the j-th text modality, and the relevant information between the visual and text modalities. The dictionary value of conditional mutual information between all visual modalities, the j-th text modal, and the relevant information relating the i-th visual modal and the j-th text modal. is the vector for each feature in the prior dictionary.

[0056] On the other hand, the visual text cross-modal encoding based on the visual text cross-modal consensus information is obtained through the following steps:

[0057] Initialize the prior dictionary;

[0058] Determine the target modality based on the task objective of the visual emotion recognition task to be processed;

[0059] Based on the sample dataset, target prior knowledge that is compatible with the target modality is selected from the prior dictionary;

[0060] Based on the prior knowledge of the target, cross-modal information fusion is performed on the visual modality and the text modality in the sample dataset to extract the visual-text cross-modal consensus information between the visual modality and the text modality;

[0061] Cross-modal fusion encoding is performed based on the visual features, text features, and target prior knowledge corresponding to the cross-modal consensus information of the visual text to obtain the cross-modal encoding of the visual text;

[0062] The target modality is one of a visual modality, a text modality, and a combined visual-text modality, and the visual feature is an image feature or a video feature.

[0063] On the other hand, when the visual emotion recognition task to be processed is a comment-based visual emotion recognition task, the target modality is a text modality;

[0064] The step of selecting target prior knowledge that matches the target modality from the prior dictionary based on the sample dataset includes:

[0065] Calculate the second conditional mutual information between the prior knowledge in the prior dictionary and the text modalities in the sample dataset;

[0066] Establish a one-to-one mapping relationship between the prior knowledge and the second conditional mutual information to obtain the second prior knowledge-conditional mutual information dictionary;

[0067] Based on the second prior knowledge-conditional mutual information dictionary values ​​in the second prior knowledge-conditional mutual information dictionary in descending order, select the third preset number of prior knowledge that has the highest correlation with the text modalities in the sample dataset;

[0068] The target prior knowledge is selected from the third preset number of prior knowledge that has the highest correlation with each text modality in the sample dataset, and the fourth preset number of prior knowledge that has the highest correlation is selected as the target prior knowledge.

[0069] On the other hand, the cross-modal information fusion of the visual modalities and text modalities in the sample dataset based on the target prior knowledge, and the extraction of the visual-text cross-modal consensus information between the visual and text modalities, includes:

[0070] Intramodal information integration and cross-modal information integration are performed on the visual modal and text modal in the sample dataset, respectively, to obtain the intramodal information integration result and the global consensus information integration result;

[0071] Based on the intramodal information integration results and the global consensus information integration results, a text cross-attention module is constructed to search for visual information related to the text modality in the visual modality based on the target prior knowledge;

[0072] Based on the intramodal information integration results and the global consensus information integration results, a visual cross-attention module is constructed to search for text information related to the visual modality in the text modality based on the target prior knowledge;

[0073] Based on visual information associated with the text modality in the visual modality, textual information associated with the visual modality in the text modality, and the prior knowledge of the target, an attention module is constructed to fuse and extract the cross-modal consensus information of the visual and textual data.

[0074] On the other hand, the process of integrating intra-modal information and cross-modal information on the visual modal and text modal in the sample dataset, respectively, to obtain intra-modal information integration results and global consensus information integration results, includes:

[0075] Intramodal information is integrated from the visual modalities in the sample dataset to obtain visual modal information;

[0076] Intramodal information is integrated from the text modalities in the sample dataset to obtain text modal information;

[0077] Intramodal information integration is performed on the aforementioned prior knowledge of the target to obtain modal information of the prior knowledge of the target;

[0078] The visual modal information, the text modal information, and the target prior knowledge modal information are used as the result of the intramodal information integration;

[0079] Based on the target prior knowledge, cross-modal information is integrated between visual modality and text modality to establish a cross-modal information association and interaction model of the visual modality information, the text modality information, and the target prior knowledge modality information;

[0080] Based on the cross-modal information association and interaction model, the transformation encoder model is invoked to integrate visual features, text features, and the target prior knowledge to obtain the global consensus information integration result.

[0081] Specifically, when processing the visual features, the text features and the target prior knowledge are masked; when processing the text features, the visual features and the target prior knowledge are masked; when processing the target prior knowledge, no masks are set for the visual features, the text features, and the target prior knowledge.

[0082] On the other hand, the construction of a text cross-attention module based on the intra-modal information integration result and the global consensus information integration result to search for visual information in the visual modality associated with the text modality based on the target prior knowledge includes:

[0083] The visual modal information and the target prior knowledge in the intramodal information integration result are merged into the key-value data of the text cross-attention module. The text modal information in the intramodal information integration result is used as the query statement of the text cross-attention module. The text cross-attention module is used to query the corresponding key-value data based on the query statement to obtain the visual information in the visual modality associated with the text modality.

[0084] The step of constructing a visual cross-attention module based on the intra-modal information integration result and the global consensus information integration result to search for text information in the text modality associated with the visual modality based on the target prior knowledge includes:

[0085] The text modal information and the target prior knowledge in the intramodal information integration result are merged into the key-value data of the visual cross-attention module. The visual modal information in the intramodal information integration result is used as the query statement of the visual cross-attention module. The visual cross-attention module is then used to query the corresponding key-value data based on the query statement to obtain the text information in the text modality associated with the visual modality.

[0086] On the other hand, the construction of an attention module based on visual information associated with the textual modality in the visual modality, textual information associated with the visual modality in the textual modality, and the target prior knowledge to fuse and extract the cross-modal consensus information between visual and textual data includes:

[0087] Visual information in the visual modality that is associated with the text modality is used as text tags, text information in the text modality that is associated with the visual modality is used as visual tags, and the target prior knowledge is used as prior knowledge tags.

[0088] After splicing the visual markers, the prior knowledge markers, and the text markers, the information is input into the attention module to fuse and extract the cross-modal consensus information of the visual text.

[0089] In the attention module, the visual tags and the prior knowledge tags are merged into key-value data, and the text tags are used as query statements. Additionally, the text tags and the prior knowledge tags are merged into key-value data, and the visual tags are used as query statements to output the visual-text cross-modal consensus information.

[0090] On the other hand, the visual features are image features;

[0091] The step of performing cross-modal fusion encoding based on the visual features, text features, and target prior knowledge corresponding to the cross-modal consensus information of the visual text to obtain the cross-modal encoding of the visual text includes:

[0092] The output results of multiple executions of the target prior knowledge adapted to the target modality by selecting from the prior dictionary and the visual-text cross-modal consensus information between the visual modality and the text modality are fused using a residual structure and then encoded to obtain the visual-text cross-modal encoding.

[0093] The output of the current iteration serves as the input data for the next iteration, which involves selecting target prior knowledge from the prior dictionary that is compatible with the target modality and extracting visual-text cross-modal consensus information between the visual and text modalities.

[0094] On the other hand, the method of using a residual structure to fuse the output results after repeatedly executing the process of selecting target prior knowledge from the prior dictionary that is compatible with the target modality and extracting the visual-text cross-modal consensus information between the visual modality and the text modality, and then encoding the results to obtain the visual-text cross-modal encoding, includes:

[0095] use The image feature encoding in the i-th output result is subjected to feature fusion processing to obtain the image feature fusion result corresponding to the i-th output result;

[0096] use The text feature encoding in the i-th output result is subjected to feature fusion processing to obtain the text feature fusion result corresponding to the i-th output result;

[0097] use The prior knowledge encoding in the i-th output result is fused to obtain the prior knowledge fusion result corresponding to the i-th output result;

[0098] By concatenating the image feature fusion result corresponding to the i-th output result, the text feature fusion result corresponding to the i-th output result, and the prior knowledge fusion result corresponding to the i-th output result, the feature fusion result corresponding to the i-th output result is obtained;

[0099] If i is not N, then for the feature fusion result corresponding to the i-th output result, the target prior knowledge adapted to the target modality is selected from the prior dictionary and the visual-text cross-modal consensus information between the visual modality and the text modality is extracted to obtain the (i+1)-th output result;

[0100] If i is N, then the feature fusion result corresponding to the i-th output result is used as the cross-modal encoding of the visual text;

[0101] in, The image feature fusion result corresponding to the output result of the i-th iteration is... The residual coefficient is... Encode the image features in the output result of the i-th iteration. For the input image features, This represents the text feature fusion result corresponding to the output result of the i-th iteration. Encode the text features in the output result of the i-th iteration. The features of the input text. The prior knowledge fusion result corresponding to the output result of the i-th iteration. Encode the prior knowledge in the output result of the i-th iteration. The input is the target prior knowledge.

[0102] On the other hand, the visual features are video features;

[0103] The step of performing cross-modal fusion encoding based on the visual features, text features, and target prior knowledge corresponding to the cross-modal consensus information of the visual text to obtain the cross-modal encoding of the visual text includes:

[0104] The visual-text cross-modal coding is obtained by fusing the target prior knowledge adapted to the target modality selected from the prior dictionary and the visual-text cross-modal consensus information extracted between the visual modality and the text modality using learnable residual parameters.

[0105] The output of the current iteration serves as the input data for the next iteration, which involves selecting target prior knowledge from the prior dictionary that is compatible with the target modality and extracting visual-text cross-modal consensus information between the visual and text modalities.

[0106] On the other hand, the process of fusing the output results after repeatedly executing the process of selecting target prior knowledge adapted to the target modality from the prior dictionary and extracting the visual-text cross-modal consensus information between the visual and text modalities using learnable residual parameters, and then encoding the results, yields the visual-text cross-modal encoding, including:

[0107] For the video features in the i-th output result, set corresponding frame coefficients for each video frame, obtain video residual block features based on the frame coefficients and the frame features of the video frames, and fuse the video residual block features with the video features in the i-th output result to obtain the video feature fusion result corresponding to the i-th output result;

[0108] For the text features in the i-th output result, set corresponding text marker coefficients for each text marker, obtain text residual block features based on the text marker coefficients and the text markers, and fuse the text residual block features with the text features in the i-th output result to obtain the text feature fusion result corresponding to the i-th output result;

[0109] For the target prior knowledge in the i-th output result, set corresponding prior knowledge label coefficients for each prior knowledge label, obtain prior knowledge residual block features based on the prior knowledge label coefficients and the prior knowledge labels, and fuse the prior knowledge residual block features with the target prior knowledge in the i-th output result to obtain the prior knowledge fusion result corresponding to the i-th output result;

[0110] By concatenating the video feature fusion result corresponding to the i-th output result, the text feature fusion result corresponding to the i-th output result, and the prior knowledge fusion result corresponding to the i-th output result, the feature fusion result corresponding to the i-th output result is obtained;

[0111] If i is not N, then for the feature fusion result corresponding to the i-th output result, the target prior knowledge adapted to the target modality is selected from the prior dictionary and the visual-text cross-modal consensus information between the visual modality and the text modality is extracted to obtain the (i+1)-th output result;

[0112] If i is N, then the feature fusion result corresponding to the i-th output result is used as the visual text cross-modal encoding.

[0113] To address the aforementioned technical problems, the present invention also provides a visual emotion recognition device, comprising:

[0114] The first acquisition unit is used to acquire the initial visual model and sample dataset;

[0115] The first training unit is used to configure a first attention head based on text sentiment evaluation coding, a second attention head based on visual text cross-modal consensus information, and a third attention head that fuses the text sentiment evaluation coding and the visual text cross-modal coding for the initial visual model. After concatenating the loss values ​​of each attention head into the model loss value of the initial visual model, the unit uses the sample dataset, the task objective of the visual sentiment recognition task to be processed, and the loss values ​​of each attention head to train the weights of each attention head in the initial visual model and output the visual sentiment recognition model.

[0116] The first computing unit is used to respond to the visual emotion recognition task to be processed by calling the visual emotion recognition model to perform emotion recognition processing on the input data to be recognized, and to obtain the visual emotion recognition result.

[0117] Specifically, according to the task objective of the visual emotion recognition task to be processed, the sample dataset includes image data with annotated text or video data with annotated text; the visual-text cross-modal consensus information is constructed through mutual information between the visual modality and the text modality.

[0118] To address the aforementioned technical problems, the present invention also provides a visual emotion recognition device, comprising:

[0119] Memory, used to store computer programs;

[0120] A processor for executing the computer program, which, when executed by the processor, implements the steps of the visual emotion recognition method as described in any of the preceding descriptions.

[0121] To address the aforementioned technical problems, the present invention also provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the visual emotion recognition method as described in any of the preceding claims.

[0122] The visual emotion recognition method provided by this invention configures a multi-head attention mechanism in the initial visual model, which includes text emotion evaluation encoding, visual-text cross-modal encoding, and fusion encoding. It constructs visual-text cross-modal consensus information through mutual information between the visual and text modalities, and configures the visual-text cross-modal encoding to be based on this consensus information. By using consensus encoding instead of alignment encoding, it reduces cross-modal encoding loss and enhances the expressive power of visual-text cross-modal encoding. The loss values ​​of each attention head are concatenated to form the model loss value. The weights of each attention head in the model are trained using the sample dataset, task objective, and the loss values ​​of each attention head, enabling the visual emotion recognition model to more accurately understand the cross-modal emotion consensus. This allows the visual emotion recognition model to execute the visual emotion recognition task, achieving more accurate visual emotion recognition. Based on the sample dataset and task objective, it can obtain more accurate recognition results in image and video emotion recognition.

[0123] The visual emotion recognition method provided by this invention further improves the accuracy of video emotion recognition by employing an attention head based on video frame text cross-modal consensus information and an attention head based on video text cross-modal consensus information in the video emotion recognition task, thereby enabling the learning of local features and global features of the video.

[0124] The visual emotion recognition method provided by this invention also provides a model loss calculation formula for a multi-head attention mechanism design that integrates text emotion evaluation encoding, video frame text cross-modal encoding based on video frame text cross-modal consensus information, video text cross-modal encoding based on video text cross-modal consensus information, and fusion encoding. Furthermore, the loss value of the attention head can be used as the emotion classification loss and opinion classification loss to train the emotion classification loss weight and opinion classification loss weight respectively, further improving the model's emotion recognition ability and thus improving the accuracy of video emotion recognition.

[0125] The visual emotion recognition method provided by this invention also extracts video features from video samples based on text features in the annotation text during the training of the visual emotion recognition model. The text features can be obtained by concatenating the subject and object in the annotation text and the search keywords obtained from the video samples. Compared with traditional video feature extraction schemes, this provides rich contextual information for the visual emotion recognition model at different levels, helping the visual emotion recognition model to better understand the video content and improve the effect of video feature extraction.

[0126] The visual emotion recognition method provided by this invention also solves the problem of insufficient understanding ability of visual emotion recognition models when processing long videos by acquiring video actions based on video segmentation information, such as by calling a video action recognition model to identify the action prediction results of segmented sub-video samples, or by calling a video description model to generate descriptive text for each sub-video sample.

[0127] The visual emotion recognition method provided by this invention also obtains an initial visual model by constructing a loss function pre-training based on visual text matching tasks, masked text prediction tasks, and consensus information filtering tasks. This enables the recognition of matching between visual samples and text samples, the recognition of prediction of masked information based on unmasked context information, and the filtering of consensus information of visual and text modalities from a priori dictionary. As a result, the training efficiency can be significantly improved when training visual emotion recognition models for different task objectives.

[0128] The visual emotion recognition method provided by this invention also offers a scheme for constructing cross-modal consensus information of visual text by filtering target prior knowledge based on a self-prior dictionary. By learning the knowledge and information contained in image or video language data into the model's prior dictionary, the model can better understand the content or contextual information of images or videos, and quickly guide the model to focus on the important parts of the image or video and its corresponding language information. Then, by performing cross-modal encoding of visual text based on cross-modal consensus information of visual text, using target prior knowledge as a bridge, the problem of weak correlation between visual and text modalities is solved. Compared with traditional cross-modal feature extraction methods, this effectively reduces the loss caused by forced semantic alignment.

[0129] The present invention also provides a visual emotion recognition device, apparatus and readable storage medium, which have the above-mentioned beneficial effects, and will not be described in detail here. Attached Figure Description

[0130] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0131] Figure 1 A flowchart of a visual emotion recognition method provided in an embodiment of the present invention;

[0132] Figure 2 A schematic diagram of a training framework for a video emotion recognition model provided in an embodiment of the present invention;

[0133] Figure 3 A flowchart of a video feature extraction method provided in an embodiment of the present invention;

[0134] Figure 4 A schematic diagram of a grounding encoder provided for an embodiment of the present invention;

[0135] Figure 5 A flowchart of video frame-text cross-modal coding provided in an embodiment of the present invention;

[0136] Figure 6 A flowchart of a video-text cross-modal coding method is provided as an embodiment of the present invention;

[0137] Figure 7 This is a schematic diagram of the structure of a visual emotion recognition device provided in an embodiment of the present invention;

[0138] Figure 8 This is a schematic diagram of the structure of a visual emotion recognition device provided in an embodiment of the present invention. Detailed Implementation

[0139] The core of this invention is to provide a visual emotion recognition method, apparatus, device, and readable storage medium to improve the visual emotion recognition capability of computer vision, and to enhance the ability to recognize emotions in images and videos.

[0140] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0141] To facilitate understanding of the technical solutions provided in the embodiments of the present invention, some key terms used in the embodiments of the present invention will be explained here first:

[0142] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0143] The Transformer model is a Natural Language Processing (NLP) model that employs a self-attention mechanism, enabling parallel training and access to global information. It primarily consists of an encoder and a decoder, each comprising six modules. The workflow involves: obtaining the representation vector of each word in the input sentence (derived by adding the word vector and its position vector), specifically inputting this word representation vector matrix into the encoder. After passing through the six encoder modules, the encoded information matrix of all words in the sentence is obtained, with the output matrix of each encoder module having the same dimensions as the input. The encoded information matrix output by the encoder is then passed to the decoder, which sequentially translates the next word i based on the currently translated word i. During this process, when translating word i+1, a masking operation is used to cover the words after i+1, and so on.

[0144] Vision-Text Matching (VTM) includes image-text matching and video-text matching. By forcing visual-text pairs to be semantically close and unpaired instances to be far apart, it learns a joint representation space, achieving cross-modal semantic alignment and cross-modal semantic propagation.

[0145] Masked Language Modeling (MLM) is a neural network-based language model that can be trained on a massive unlabeled corpus during the pre-training phase and then fine-tuned in supervised tasks such as text classification and sequence labeling.

[0146] A fully connected layer (FC) is a term from convolutional neural networks (ConvNet or CNN). In a CNN architecture, after multiple convolutional and pooling layers, one or more fully connected layers are connected. Similar to a multilayer perceptron (MLP), each neuron in a fully connected layer is fully connected to all neurons in the layer preceding it. Fully connected layers can integrate class-discriminating local information from convolutional or pooling layers. To improve CNN network performance, the activation function of each neuron in a fully connected layer typically uses the Rectified Linear Unit (ReLU) function. The output of the last fully connected layer is passed to an output function, which can be classified using softmax regression. This layer can also be called a softmax layer. For a specific classification task, choosing a suitable loss function is crucial. Convolutional neural networks have several commonly used loss functions, each with its own characteristics. Typically, the fully connected layers of a convolutional neural network have the same structure as those of a multilayer perceptron, and the training algorithm for convolutional neural networks often employs the error back propagation (BP) algorithm.

[0147] The Contrastive Language-Image Pretraining (CLIP) model is a pre-trained model designed to learn the relationships between images and text for text-image retrieval and other related applications. The CLIP model comprises two main modalities: a text modality and a visual modality, processed by a text encoder and an image encoder, respectively. Both modalities output fixed-length vector representations, or embeddings. During training, the CLIP model uses a large number of text-image pairs from the internet as training data. Each text-image pair is treated as a positive sample because it is a pair, while the other corresponding images are treated as negative samples. This design allows the CLIP model to learn high-level semantic features between images and text, rather than relying solely on pixel-level supervision information.

[0148] Bidirectional Encoder Representations from Transformers (BERT) is a pre-trained language representation model that emphasizes a departure from traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training. Instead, it employs a novel masked language modeling (MLM) to generate deep bidirectional language representations.

[0149] The grounding encoder is the encoder in the grounding language model. Grounding connects the language model with the specific environment (data / API / service / physical world, etc.), which is fundamental to solving many practical tasks.

[0150] With the continuous development and innovation of computer vision technology, cross-modal learning between images or videos and corresponding text modalities based on visual models can enable computer vision to recognize the emotions carried by images or videos, thereby performing tasks such as emotion classification of images or videos, question answering based on images or videos, and retrieval and recommendation of images or videos.

[0151] On social media platforms, users post comments while viewing images or videos. These comments carry rich emotional information and are an important means of information transmission on social media platforms. Traditional visual-text cross-modal learning models extract features corresponding to the visual and textual modalities through visual and language backbone networks and input them into different types of transformational encoder models. However, due to the weak correlation between the two modalities and the semantic differences between them, forcibly associating the visual and textual modalities will reduce the model's cross-modal capability to some extent. Current computer vision-based solutions for understanding the emotions in images or videos lack comment-based cross-modal learning solutions. Comments often have a stronger subjective element than the descriptive text accompanying images or videos. For example, some comments contain elusive emotions or subtle feelings, such as sarcasm or innuendo. Furthermore, different users have different focuses, leading to a weak correlation between the comment content and the semantics of the image or video. In addition, the short length of comments results in a lack of sufficient contextual information, further increasing the difficulty of sentiment analysis.

[0152] In response to these issues, the visual emotion recognition method provided in this invention aims to offer an effective solution for analyzing the emotional information of visual modalities based on comments, addressing problems such as the current mainstream visual models forcibly associating visual and text modalities, the different focus ranges of text modalities on visual modalities, and the weak correlation between visual and text modalities.

[0153] In terms of system architecture, the visual emotion recognition method provided in this embodiment of the invention can be implemented based on a single computing device or a computing system composed of multiple computing devices. The computing device used can be, but is not limited to, a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and a data processing unit (DPU), or one or more of these as accelerators, or other types of computing devices.

[0154] The visual emotion recognition method provided in this embodiment of the invention can be further applied to a visual emotion recognition system comprising a computing system, a storage system, and a client device. The storage system stores sample datasets for training the visual emotion recognition model. The computing system reads the sample datasets from the storage system and executes the training task of the visual emotion recognition model according to the task objective of the visual emotion recognition task to be processed. The client device responds to the visual emotion recognition task output by the user, parses the data to be recognized and its related video emotion recognition task parameters, and transmits them to the computing system. The computing system further responds to the visual emotion recognition task by calling the visual emotion recognition model to perform emotion recognition processing on the input data to be recognized, obtaining the visual emotion recognition result. The client device then displays the visual emotion recognition result in the form required by the visual emotion recognition task to be processed.

[0155] The visual emotion recognition method provided in this invention can be used to solve image emotion recognition tasks and video emotion recognition tasks, such as recognizing the emotional reactions of a group of users to specific content triggered by images or videos posted on social platforms based on comments.

[0156] Based on the above architecture, the visual emotion recognition method provided by the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0157] Figure 1 This is a flowchart of a visual emotion recognition method provided in an embodiment of the present invention.

[0158] like Figure 1 As shown, the visual emotion recognition method provided in this embodiment of the invention may include:

[0159] S101: Obtain the initial visual model and sample dataset.

[0160] S102: Configure the initial visual model with a first attention head based on text sentiment evaluation encoding, a second attention head based on visual text cross-modal consensus information, and a third attention head that integrates text sentiment evaluation encoding and visual text cross-modal encoding. After concatenating the loss values ​​of each attention head into the model loss value of the initial visual model, use the sample dataset, the task objective of the visual emotion recognition task to be processed, and the loss values ​​of each attention head to train the weights of each attention head in the initial visual model, and output the visual emotion recognition model.

[0161] S103: In response to the visual emotion recognition task to be processed, the visual emotion recognition model is invoked to perform emotion recognition processing on the input data to be recognized, and the visual emotion recognition result is obtained.

[0162] Specifically, based on the task objective of the visual emotion recognition task to be processed, the sample dataset includes image data or video data with annotated text; visual-text cross-modal consensus information is constructed through mutual information between the visual modality and the text modality.

[0163] In this embodiment of the invention, for S101, depending on whether the task objective of the visual emotion recognition task to be processed is image emotion recognition or video emotion recognition, a corresponding initial visual model is obtained as an initial image language model or an initial video language model. The initial visual model can be a general visual understanding model in traditional visual models, or it can be trained based on other sample data. Depending on whether the task objective of the visual emotion recognition task to be processed is image emotion recognition or video emotion recognition, a corresponding image-text sample set or video-text sample set is obtained. The text samples contain comments on the images or videos.

[0164] For S102, traditional visual models generally perform poorly in downstream tasks such as visual emotion recognition. To improve the accuracy of visual emotion recognition, further parameter tuning of the initial visual model is required.

[0165] Because comments often contain strong emotional information—these are users' emotional responses to the video content, such as "This is so funny" or "This is so sad"—a first attention head based on text sentiment evaluation encoding is configured for the initial visual model. Due to semantic misalignment between the visual and text modalities, a second attention head based on visual-text cross-modal consensus information (visual-text cross-modal encoding) is configured for the initial visual model. Since the emotional nuances of the visual modality cannot always be derived visually and require specific context, as well as text evaluation to help the model understand elusive visual information and key content within the visual modality, a third attention head fusing text sentiment evaluation encoding and visual-text cross-modal encoding is configured for the initial visual model, in addition to the second attention head based on visual-text cross-modal consensus information.

[0166] In some optional embodiments of the present invention, the attention header is composed of a fully connected layer, such as... Where X is the input feature and Y is the final output class probability. The input features are transformed to a specified dimension, and then passed through the Rectified Linear Unit (ReLU) activation function of the neural network. Map from a specified dimension to a category.

[0167] This invention reduces cross-modal encoding loss and enhances the expressive power of visual-text cross-modal encoding by configuring a second attention head based on visual-text cross-modal consensus information for the initial visual model, rather than using aligned encoding. The visual-text cross-modal consensus information is constructed through mutual information between the visual and text modalities. It connects high-level semantic information between the visual and text modalities and can be obtained by filtering from either the visual or text modal perspective, or from a combined visual-text modal perspective.

[0168] In comment-based visual sentiment recognition tasks, since comment content often dominates visual sentiment analysis, the visual text cross-modal encoding based on visual text cross-modal consensus information can be configured to filter visual text cross-modal consensus information based on text modality. This makes the filtered visual text cross-modal consensus information closer to the text modality, which is more conducive to visual sentiment analysis.

[0169] When training a visual emotion recognition model, the model learns to predict the uncertainty of the task and automatically discovers the optimal trade-off between tasks, reducing the need for manual adjustment of hyperparameters between branches and tasks. This can be achieved by training the weights of different attention heads, thus optimizing the visual emotion recognition model from the initial visual model.

[0170] For S103, for images or videos on social media platforms, the system collects images and their accompanying comments, or videos and their accompanying comments. This data is then input into a visual emotion recognition model for text-based emotion evaluation coding, visual-text cross-modal coding based on visual-text cross-modal consensus information, and fusion coding. The resulting emotion recognition result is then calculated. The emotion recognition result can be positive or negative, or further subdivided into different categories within positive or negative emotions. Alternatively, different emotions can be associated with corresponding visual content, such as expressing longing for a landscape image, excitement about a basketball game video, or anger about one or more video frames.

[0171] The visual emotion recognition method provided in this invention configures a multi-head attention mechanism in the initial visual model, which includes text emotion evaluation encoding, visual-text cross-modal encoding, and fusion encoding. It constructs visual-text cross-modal consensus information through mutual information between the visual and text modalities, and configures the visual-text cross-modal encoding to be based on this consensus information. By using consensus encoding instead of alignment encoding, it reduces cross-modal encoding loss and enhances the expressive power of visual-text cross-modal encoding. The loss values ​​of each attention head are concatenated to form the model loss value. The weights of each attention head in the model are trained using the sample dataset, task objective, and the loss values ​​of each attention head, enabling the visual emotion recognition model to more accurately understand the cross-modal emotion consensus. This allows the visual emotion recognition model to execute the visual emotion recognition task, achieving more accurate visual emotion recognition. Based on the sample dataset and task objective, more accurate recognition results can be obtained in image and video emotion recognition.

[0172] Figure 2 This is a schematic diagram of the training framework for a video emotion recognition model provided in an embodiment of the present invention.

[0173] In some optional embodiments of the present invention, the visual emotion recognition task to be processed is further described as a video emotion recognition task, and the visual emotion recognition model to be trained is a video emotion recognition model. When the visual emotion recognition task to be processed is a video emotion recognition task, the second attention head may include a fourth attention head based on video frame text cross-modal coding based on video frame text cross-modal consensus information and a fifth attention head based on video text cross-modal coding based on video text cross-modal consensus information.

[0174] Please refer to Figure 2 Four attention heads can be set, including: a first attention head based on text sentiment evaluation encoding (which can be implemented based on a language backbone network). Figure 2The attention head, designated as attention head 0, is used to learn evaluations with strong emotional information in text evaluations and can be obtained through a language backbone network; the fourth attention head is based on cross-modal coding of video frame text using video frame text consensus information. Figure 2 The attention head, designated as attention head 1, is used to learn the association and alignment between text and video frames, and even short videos. It can be obtained through a video frame-text cross-modal encoder. The fifth attention head for video text cross-modal encoding is based on video text cross-modal consensus information. Figure 2 The attention head, designated as attention head 2, is used to learn the association and alignment between text evaluation and the entire video, and can be obtained through a video-text cross-modal encoder; the third attention head is a fusion of text sentiment evaluation encoding and visual text cross-modal encoding. Figure 2 The attention head (numbered 3) is used to fuse the three different types of visual and linguistic information mentioned above, further helping the model to associate and align visual and linguistic textual information. It can be obtained by transforming the encoder model.

[0175] For video sentiment recognition tasks, since text evaluation often dominates sentiment analysis, the method for filtering cross-modal consensus information between visual and text is configured to filter cross-modal consensus information between visual and text based on the text modality, such as... Figure 2 As shown, this allows for the filtering of visual text cross-modal consensus information that is closer to the text modality, which is more beneficial for video sentiment analysis.

[0176] The video emotion recognition model is trained using a multi-task learning approach that dynamically adjusts the four attention heads mentioned above. The model loss value can be expressed by the following formula:

[0177] ;

[0178] in, Let i be the model parameters for the i-th attention head. Let i be the weight of the i-th attention head. Let i be the loss value of the i-th attention head. The model loss value; i = 0, 1, 2, 3;

[0179] Attention heads include the first attention head, the fourth attention head, the fifth attention head, and the third attention head. The first attention head can be denoted as the 0th attention head. (Attention head 0), the fourth attention head can be recorded as the first attention head. (Attention Head 1), the fifth attention head can be remembered as the second attention head. (Attention head 2), the third attention head can be recorded as the 3rd attention head. (Attention to head 3).

[0180] The weights of each attention head can be set to satisfy the requirements. You can also set It can be designed. , It is a fully connected layer. This refers to activation functions, such as the Rectified Linear Unit (ReLU) activation function in neural networks. This is a regression function. The output loss values ​​of the four attention heads are concatenated, i.e. .

[0181] Building upon this, the weights of the attention head can be further divided into sentiment classification weights and opinion classification weights to meet diverse sentiment recognition needs. The loss value of the attention head can then be expressed by the following formula:

[0182] ;

[0183] The output of the visual emotion recognition model is represented by the following formula:

[0184] ;

[0185] in, , , , ;

[0186] in, For sentiment classification loss weights, Weighting loss for opinion categories For sentiment classification loss value, Loss value for opinion classification, Here, L is the loss weight calculation function, and L is the concatenated value of the sentiment classification loss and the opinion classification loss. It is a fully connected layer. For activation function, For regression function, For the sentiment classification results, For the results of opinion classification, This refers to the model calculation results output by the video emotion recognition model based on the input video and text data.

[0187] In the initial visual model, you can set... , , , It is 0.25. , The weight is set to 0.5. The model learns appropriate weights based on the sample dataset, the task objective of the visual emotion recognition task, and the model's loss function. The final emotion classification and opinion classification results output by the video emotion recognition model are as follows:

[0188] ;

[0189] in, For the sentiment classification results, For the opinion classification results, For video emotion recognition models, based on the input video data to be recognized ( ) and the comment data it carries ( The output result.

[0190] Then as Figure 2 As shown, when the visual emotion recognition task to be processed is a video emotion recognition task, this embodiment of the invention provides a training framework for a visual emotion recognition model. This framework involves extracting frame visual features (0,1,...N) from video frames using a video frame feature extraction module, extracting text features (0,1,...M) from text samples using a language backbone network, performing video frame-to-text cross-modal encoding based on the frame visual and text features using a video frame-to-text cross-modal encoder, extracting video features based on the frame visual and text features using a video feature extraction module, performing video-to-text cross-modal encoding based on the video and text features using a video-to-text cross-modal encoder, and finally training and outputting the video frame-to-text cross-modal encoding and the video-to-text cross-modal encoding using a transformation encoder model.

[0191] When the visual emotion recognition task to be processed is an image emotion recognition task, and Figure 2 The difference is that only one second attention head can be set for image-text cross-modal coding.

[0192] The visual emotion recognition method provided in this embodiment of the invention further improves the accuracy of video emotion recognition by employing an attention head based on video frame text cross-modal consensus information and an attention head based on video text cross-modal consensus information in the video emotion recognition task, thereby enabling the learning of local features and global features of the video.

[0193] The visual emotion recognition method provided in this invention also provides a model loss calculation formula for a multi-head attention mechanism design that integrates text emotion evaluation encoding, video frame text cross-modal encoding based on video frame text cross-modal consensus information, video text cross-modal encoding based on video text cross-modal consensus information, and fusion encoding. Furthermore, the loss value of the attention head can be used to train the emotion classification loss weight and opinion classification loss weight respectively, thereby further improving the model's emotion recognition ability and thus improving the accuracy of video emotion recognition.

[0194] Figure 3 This is a flowchart of a video feature extraction method provided in an embodiment of the present invention.

[0195] Based on the above embodiments, this invention describes methods for extracting visual and textual features.

[0196] In training an image-based emotion recognition model, the visual features to be extracted are image features. In training a video-based emotion recognition model, the visual features to be extracted are frame-based visual features or video features.

[0197] A video can be viewed as a set of images along the time dimension. This embodiment of the invention first describes the steps for extracting frame visual features.

[0198] like Figure 3 As shown, the extraction of frame visual features mainly consists of two steps: frame splitting and extracting frame visual features. For the frame splitting step, a uniform sampling method can be used to select N frames from the video. The sampling result for a single video is denoted as... Where N is the number of samples, This is the result after frame extraction from the current video.

[0199] For the step of extracting frame visual features, the visual features of the visual modality can be denoted as... ,in This section describes the visual features of all frames in a single video. The selected video frames are images, and corresponding image features can be extracted using an image-related backbone network (vision backbone), such as ResNet or ViT. Here, we take ViT from the contrastive language-image pre-trained model as an example to extract visual features. The image is divided into non-overlapping 16x16 patches. These two-dimensional patches are then linearly mapped to one dimension, and corresponding position embeddings are superimposed. A 12-layer transformer encoder integrates contextual information through its self-attention mechanism to model global information and achieve feature interaction and fusion. Subsequently, a multilayer perceptron (MLP) is used to extract image features, while simultaneously promoting the transfer of feature information between different locations. The final output image feature dimension is 197x768, where 197 = 196 + 1, 196 is the patch sequence length, 1 represents special characters (classification markers cls), and 768 is the patch dimension.

[0200] It should be noted that, in Figure 3 In this model, the visual features (0, 1, ... N) and text features (0, 1, ... M) output by each layer encoder are represented by the same triplet graphical representation. This does not mean that the features output by each layer encoder are the same, but only that the triplet data indicates whether the feature is a visual feature or a text feature.

[0201] At this point, the visual features of each individual video frame can be obtained. The steps for extracting visual features of image modalities can refer to the steps for extracting frame visual features of individual video frames.

[0202] In videos, there are often relationships between video frames, such as different steps representing an action. Based on the above steps, a self-attention mechanism can be used to achieve interaction between video frames, ultimately outputting a representation of the video frames. Specific steps may include:

[0203] use Adjust the dimensions of the frame visual features, where A fully connected layer with an input dimension of 768 and an output dimension of 1024. To reduce feature dimensions Convert to FC is a fully connected layer with an input dimension of 1024 and an output dimension of 1024. Number of frames per video;

[0204] Video features are extracted using a transformer encoder, i.e. ,in This is a self-attention module with a hidden layer dimension of 1024, a depth of 4, a header of 8, and a block (path_size) size of 6×10. The final output dimension is , where b is the batch size. Number of frames per video.

[0205] When training a video emotion recognition model, it is often necessary to understand the entire video content, so it cannot be limited to frame-level features. In some optional embodiments of the present invention, when the visual emotion recognition task to be processed is a video emotion recognition task, step S102, which uses the sample dataset, the task objective of the visual emotion recognition task to be processed, and the loss values ​​of each attention head to train the weights of each attention head in the initial visual model and output the visual emotion recognition model, may include:

[0206] For video samples in the sample dataset, multiple video frames are extracted from the video samples, the frame visual features of the video frames are obtained, and time dimension information is added to the frame visual features according to the order of the video frames in the video samples.

[0207] Extract text features based on the annotation text corresponding to the video samples;

[0208] After encoding the visual and textual features of the frames with added time dimension information, the video features of the video samples are obtained.

[0209] The video features are input into the initial visual model, and the weights of each attention head in the initial visual model are adjusted according to the loss value of each attention head, and the visual emotion recognition model is output.

[0210] When training a visual emotion recognition model for video emotion recognition tasks, the required sample dataset includes video samples and their corresponding text samples. The sample dataset can be obtained from publicly available datasets, such as video-text retrieval, action classification and recognition, video description, or user-provided sample datasets for the visual emotion recognition task.

[0211] The video-to-article retrieval sample dataset may include MSR-VTT, where each video has up to 20 descriptions and a corresponding category, such as music. This type of dataset uses the corresponding category as the hashtag for this scheme, while the video description and the video itself remain unchanged.

[0212] Action classification and recognition sample datasets can include UCF101, which contains video clips for 101 action categories. Each video provides the start and end times for the action category. This type of dataset uses the action category as the hashtag for this approach. Since such datasets lack corresponding descriptions, multiple descriptions for the current video can be generated based on existing video description models, such as Vid2Seq, according to the video content, action category, and the start and end times of the action.

[0213] The video description sample dataset may include MSVD, where each video clip is accompanied by multiple manually generated English descriptions, totaling approximately 70,000 descriptions. These descriptions are designed to accurately describe the events and scenes occurring in the video clips. Since such datasets lack corresponding categories, corresponding categories can be directly generated using action classification and recognition models, such as TSN, BSN, and R2Plus1D. Then, categories with similarity are removed based on the video descriptions, and the final categories are used as the hashtags employed in this embodiment of the invention.

[0214] In order to make the video content more relevant to the text or task and to accelerate task convergence, this embodiment of the invention will adopt a cross encoder based on hashtags.

[0215] After extracting frame visual features using the method for extracting video frame visual features provided in this embodiment of the invention, since video frame features are only visual features in the spatial dimension and lack information in the temporal dimension, it is necessary to add temporal dimension information to the frame visual features in order to express the video features. Temporal dimension information includes relative time dimension and absolute time dimension. The relative time dimension is added sequentially according to the order in which the frames were extracted, while the absolute time dimension is the position of the current frame in the entire video, usually accurate to milliseconds. This solution only needs to understand the video content and does not require the absolute time dimension. Then, the video features are further extracted using the hashtag-based cross encoder provided in this embodiment of the invention.

[0216] Here, it is necessary to explain the hashtags used in training the visual emotion recognition model in this embodiment of the invention. In social media platforms, hashtags, also known as aggregation tags, typically use the hash symbol "#" or similar words or phrases to represent data categories and are used to organize topic discussions. In this embodiment of the invention, hashtags are defined as a guide for extracting video features, helping the model provide more contextual information, enhancing the joint representation learning ability of visual and linguistic modalities, and thus improving the model's generalization ability.

[0217] In some optional embodiments of the present invention, extracting text features based on the annotation text corresponding to the video sample may include: constructing a text sample corresponding to the video sample based on the annotation text and the topic tags of the video sample, and then extracting the text features corresponding to the video sample from the text sample. The topic tags may include, but are not limited to, search keywords used when obtaining the video sample or video segmentation information carried by the video sample.

[0218] When crawling videos, the search is conducted using keywords. The search results are sorted from highest to lowest relevance. Data collectors typically gather highly relevant videos, meaning these keywords are strongly correlated with the video content. For example, searching for "basketball" will return videos related to basketball.

[0219] Video segmentation information is the information that divides a video into chapters, especially in longer videos. For example, if a video is divided into segments of ten minutes each, and each segment corresponds to a chapter of the video, then the relative time position of "ten minutes" in the video is the video segmentation information.

[0220] If search keywords are used as topic tags, in some optional embodiments of this invention, extracting text features based on the annotation text corresponding to the video sample may include:

[0221] Extract the subject and object from the annotation text corresponding to the video sample;

[0222] The text features are obtained by concatenating the subject and object with the search keywords obtained from the video samples.

[0223] Video content typically expresses a viewpoint or describes a fact, and this content usually has a corresponding object and subject. It is text-oriented and targets a single video, such as "a little boy walking his dog in the park".

[0224] Then as Figure 3 As shown, assuming the annotation text (comment) of the video sample is "Yao and McGrady performed the best in the basketball team", and the search keyword for the video sample is "basketball", syntactic analysis is performed on the annotation text (comment), extracting the subject and object as "Yao" and "McGrady" respectively. Using "basketball" as the topic tag, the sentence "[classification tag][basketball][Yao][McGrady][sequence termination tag][complete character]" is obtained, where "[classification tag]" is represented by "cls", "[sequence termination tag]" is represented by "eos", and "[complete character]" is represented by "pad". The concatenated sentence is input into the language backbone network for further extraction of text features 0, 1, ..., M.

[0225] If video segmentation information is used as topic tags, text features can be extracted based on the annotation text corresponding to the video samples, including:

[0226] If a video sample corresponds to video segmentation information and segmentation event descriptions, then the video sample is divided into multiple sub-video samples according to the video segmentation information, and the corresponding segmentation event descriptions are used as annotation text for the sub-video samples.

[0227] The video action recognition model is invoked to identify and predict the actions of sub-video samples.

[0228] After extracting video description keywords from the annotation text of the sub-video sample, the action prediction results of the sub-video sample are concatenated with the video description keywords of the sub-video sample to obtain the text features of the sub-video sample.

[0229] The text features of each sub-video sample are integrated into the text features of the video sample.

[0230] Extracting video description keywords from the annotation text of sub-video samples can include: extracting video description keywords from the annotation text of sub-video samples by using a natural language processing (NLP) model to extract core keywords.

[0231] If video segmentation information is used as topic tags, and text features are extracted based on the annotation text corresponding to the video samples, the following are also included:

[0232] If a video sample corresponds to video segmentation information and segmentation action description, then the video sample is divided into multiple sub-video samples according to the video segmentation information;

[0233] The video description model is invoked to generate descriptive text for each sub-video sample;

[0234] After extracting action keywords from the segmented action description and video description keywords from the description text, the action keywords and video description keywords of the sub-video samples are concatenated to obtain the text features of the sub-video samples.

[0235] The text features of each sub-video sample are integrated into the text features of the video sample.

[0236] Extracting video description keywords from self-description text can include: extracting video description keywords from self-description text by using a natural language processing (NLP) model to extract core keywords.

[0237] In some alternative embodiments of the present invention, extracting text features based on the annotation text corresponding to the video sample may include:

[0238] If the video sample does not include video segmentation information, the subject and object are extracted from the annotation text corresponding to the video sample; the subject and object are then concatenated with the search keywords used to obtain the video sample to obtain text features.

[0239] If a video sample corresponds to video segmentation information and segment event descriptions, the video sample is divided into multiple sub-video samples based on the video segmentation information, and the corresponding segment event descriptions are used as annotation text for the sub-video samples. The video action recognition model is called to identify and obtain the action prediction results of the sub-video samples. After extracting video description keywords from the annotation text of the sub-video samples, the action prediction results of the sub-video samples are concatenated with the video description keywords of the sub-video samples to obtain the text features of the sub-video samples. The text features of each sub-video sample are integrated into the text features of the video sample.

[0240] If a video sample corresponds to video segmentation information and segmented action descriptions, the video sample is divided into multiple sub-video samples based on the video segmentation information; the video description model is called to generate description text for each sub-video sample; action keywords are extracted from the segmented action descriptions, and video description keywords are extracted from the description text; the action keywords and video description keywords of the sub-video samples are concatenated to obtain the text features of the sub-video samples; the text features of each sub-video sample are integrated into the text features of the video sample.

[0241] The next step involves encoding the visual and textual features of the frames with added temporal dimension information to obtain the video features of the video samples. For example... Figure 3 As shown, the steps for extracting video features based on hashtags and annotation text using the hashtag-based cross encoder provided in this embodiment of the invention include:

[0242] Obtain the topic tags corresponding to the video samples, as well as the subjects and objects in the annotation text. The topic tags can be obtained directly from the annotation file, while the subjects and objects in the annotation text are determined through syntactic analysis and dependency relations. For example, given the input text, syntactic analysis is first performed to determine the structure and components of the sentences. The given text is then decomposed into phrases, sentences, and words through syntactic analysis, and the relationships between them are determined.

[0243] Based on the syntactic analysis results, dependency relations are determined, and the subject or object in the annotated text is obtained; by analyzing the dependency relations in the sentence, the relationship between the verb and other components is determined, and the components that are directly or indirectly dependent on the verb are found, thereby determining the subject and object of the sentence;

[0244] The subject and object of hashtags and annotation text are concatenated, and features are extracted based on the language backbone network. Let the number of hashtags be n, the number of subjects in the comment text be m, and the number of objects be l, where n + m + l < 10. The concatenation process can include: concatenating the hashtags and the subjects and objects in the comment text. The concatenated result is denoted as . The language backbone model is selected to tokenize the text features and extract the vector (embedding). The maximum length is set to 10. If the length is less than 10, padding is used. Then, the preceding and following classification labels (cls) and sequence termination labels (eos) are added, so that the output dimension is 12x768.

[0245] Visual features of spliced ​​frames With text features ( The video features are then input into a hashtag-based cross encoder to obtain video features, which may include:

[0246] Visual features of spliced ​​frames With text features ( First, add the corresponding position codes sequentially according to the order of the video frames. As a time dimension of video frame information, additional video type encoding is added. (For example, if the data is video data, it is represented by 0; if the data is text data, it is represented by 1.) In this way, the visual features of each video frame are represented as frame feature triples. (correspond Figure 3 Visual feature triplet data); based on hashtags and subject-object pairs. Add position encoding and type encoding This represents each text feature as a text feature triple. (correspond Figure 3 (Chinese text feature triples data); concatenate frame feature triples with text feature triples to obtain input features. ,in For frame feature triples For text feature triples, N and M are the number of frames and the text length, respectively;

[0247] Visual features of the stitched frames With text features ( The input is fed into a hashtag-based cross encoder for encoding, and then passed through a multilayer perceptron to obtain video features. The hashtag-based cross encoder can consist of a four-layer transformation encoder model with a hidden layer dimension of 512, four cross-attention heads, and a drop rate of 0.1. Weight initialization can utilize a text encoder from a contrastive language-image pre-trained model. To fully leverage the knowledge of the image-text pre-trained model, a residual structure is used between visual features and cross-modal representations. ,in For pooling layers, The weights are used to obtain the final video features through a multilayer perceptron containing multiple fully connected layers. The final dimension is [b, 128], where b is the batch size and 128 is the output dimension of the multilayer perceptron. The visual emotion recognition method provided in this embodiment of the invention also extracts video features from video samples based on text features in the annotation text during the training of the visual emotion recognition model. This can be achieved by concatenating the subject and object in the annotation text with the search keywords used to obtain the video samples. Compared to traditional video feature extraction schemes, this provides rich contextual information for the visual emotion recognition model at different levels, helping the model better understand video content and improving the effectiveness of video feature extraction.

[0248] The visual emotion recognition method provided in this embodiment of the invention also obtains video actions based on video segmentation information from long videos. For example, it obtains the action prediction results of the segmented sub-video samples by calling a video action recognition model, or generates descriptive text for each sub-video sample by calling a video description model. This solves the problem of insufficient understanding ability of visual emotion recognition models when processing long videos.

[0249] To further improve the generalization ability of the visual emotion recognition model, this embodiment of the invention provides a pre-training scheme for an initial visual model as a general visual model.

[0250] In this embodiment of the invention, the initial visual model is obtained through the following steps:

[0251] Based on visual text matching, masked text prediction and consensus information filtering tasks, a loss function is constructed, and an initial visual model is obtained through pre-training.

[0252] Among them, the visual text matching task is used to identify the matching situation between visual samples and text samples, the masked text prediction task is used to identify the information to be masked based on the context information that is not masked, and the consensus information filtering task is used to filter the consensus information of visual modal and text modal from the prior dictionary.

[0253] Training the initial visual model also requires a sample dataset. The sample dataset can be collected using the same method employed in training the visual emotion recognition model. It should be noted that, to adapt to more downstream tasks, the initial visual model can use a more general sample dataset, such as only including image samples and their corresponding text samples, or video samples and their corresponding text samples. The text samples are not limited to comments; they can also be annotation text inherent to the image or video samples. This annotation text can include, but is not limited to, titles, summaries, subtitles, or other descriptive text from the image or video, and can also include text information identified from the image or video using an image recognition model or video recognition model.

[0254] In this embodiment of the invention, an initial visual model is pre-trained by fusing visual and linguistic information to achieve cross-modal understanding and generation. This allows for the large-scale utilization of unlabeled data through unsupervised learning to learn rich visual and linguistic knowledge. The training framework for the initial visual model can also be referenced. Figure 2 The provided training framework for a video emotion recognition model inputs the output of a transformation encoder model to a visual-text matching model and a masked language model to perform visual-text matching and masked text prediction tasks. By setting three loss functions, it achieves matching of visual and textual modalities, masked prediction, and acquisition of cross-modal consensus information.

[0255] In this embodiment of the invention, the visual-text matching task is used to identify the matching status between visual samples and text samples. For example, a match can be set to 1, and a non-match to 0. Taking the three sample datasets listed in the above embodiments of the invention for training the visual emotion recognition model as examples, in the video-text matching task, for video-text retrieval and video description, since there are no similar videos, other videos or the text corresponding to other videos are randomly selected during matching; while for action classification and recognition datasets, there are similar phenomena in different videos, so it is necessary to randomly select from other types of videos or the text corresponding to other videos. The loss function corresponding to the visual-text matching task can be:

[0256] ;

[0257] in, Let S be the loss value for the visual-text matching task, and S be the number of matching pairs between the visual and text modalities. For sign functions, when the visual modality matches the text modality A value of 1 indicates a mismatch between the visual and textual modalities. =0, For the i-th visual modality, For the i-th text modality, The probability of matching visual modalities with text modalities.

[0258] In this embodiment of the invention, the masked text prediction task is used to identify and predict masked information based on unmasked context information. Unsupervised pre-training is implemented using masked language modeling (MLM) to predict the masked words based on the remaining unmasked context information. Specifically, it involves masking a subset of words in the text, predicting the masked portion based on input image or video data and other unmasked words.

[0259] Based on the embodiments of the present invention described above, when extracting visual features, the context of the visual modality, excluding the annotation text, can be enriched by constructing topic tags. Using topic tags as visual tags, the masked text prediction task falls into two categories: the first involves masking the visual tags and predicting the masked visual tags based on the input visual and text data; the second involves masking the annotation text and predicting the masked annotation text based on the input visual data and topic tags.

[0260] The loss function for the masked text prediction task can be:

[0261] ;

[0262] ;

[0263] in, Let S be the first mask text prediction loss value, S be the number of matching pairs between the visual modality and the text modality, and V be the dictionary size of the text modality. Let be the sign function, which is defined when the visual label predicted based on the v-th visual modality matches the masked visual label in the i-th text modality. The value is 1 when the visual label predicted based on the v-th visual modality does not match the masked visual label in the i-th text modality. =0, For the i-th visual modality, For the i-th text modality, To predict the probability of a masked visual label based on the input visual and textual modalities;

[0264] The loss value for predicting the second masked text. Let be the sign function, which is defined when the annotation text predicted based on the v-th visual modality matches the masked annotation text in the i-th text modality. The value is 1 when the annotation text predicted based on the v-th visual modality does not match the masked annotation text in the i-th text modality. =0, Let i be the visual label for the i-th visual modality. Let be the masked annotation text in the i-th text modality. To predict the probability of a masked text modality based on the input visual modality, visual label, and remaining text modality.

[0265] When training the initial visual model, the masking language model can perform 15% masking.

[0266] In this embodiment of the invention, the consensus information filtering task is used to filter consensus information from the prior dictionary for both the visual and textual modalities. In some implementations of this invention, a new dictionary can be constructed to enable the model to learn cross-modal consensus information from the visual to the textual modal. By randomly initializing the prior dictionary and iteratively updating it during the pre-training of the initial visual model, the model learns the cross-modal consensus information into the dictionary. The loss function corresponding to the consensus information filtering task can be:

[0267] ;

[0268] in, The loss function for the consensus information filtering task is defined, where K is the amount of prior knowledge related to the target. For vectorized functions, For weight parameters, For the i-th visual feature, For the j-th text feature, To associate relevant information between the i-th visual modality and the j-th text modality, Let be the dictionary value of conditional mutual information between the i-th visual modality, the j-th text modality, and the relevant information relating the i-th visual modality and the j-th text modality. For information between visual and textual modalities, Let be the conditional mutual information dictionary values ​​for the i-th visual modality, the j-th text modality, and the relevant information between the visual and text modalities. The dictionary value of conditional mutual information between all visual modalities, the j-th text modal, and the relevant information relating the i-th visual modal and the j-th text modal. is the vector for each feature in the prior dictionary.

[0269] In the loss function corresponding to the consensus information filtering task, the first term is the L2 loss function (also known as the Euclidean loss function or the mean squared error (MSE) loss function), which makes... The first term maps to the target prior knowledge representation; the second term is also an L2 loss function, used to constrain the vector (embedding) space of the prior dictionary to prevent it from growing too fast.

[0270] The visual emotion recognition method provided in this invention also obtains an initial visual model by constructing a loss function pre-training based on visual text matching tasks, masked text prediction tasks, and consensus information filtering tasks. This enables the recognition of matching between visual samples and text samples, the recognition of prediction of masked information based on unmasked context information, and the filtering of consensus information of visual and text modalities from a priori dictionary. As a result, the training efficiency can be significantly improved when training visual emotion recognition models for different task objectives.

[0271] Based on the above embodiments, the present invention further describes the steps for establishing cross-modal consensus information for visual text.

[0272] In this embodiment of the invention, the visual text cross-modal encoding based on visual text cross-modal consensus information in S102 can be obtained through the following steps:

[0273] Initialize the prior dictionary;

[0274] Determine the target modality based on the task objective of the visual emotion recognition task to be processed;

[0275] Based on the sample dataset, target prior knowledge that is compatible with the target modality is selected from the prior dictionary.

[0276] Based on prior knowledge of the target, cross-modal information fusion is performed on the visual modality and the text modality in the sample dataset to extract cross-modal consensus information between the visual modality and the text modality;

[0277] Cross-modal fusion encoding is performed based on the visual features, text features, and target prior knowledge corresponding to the cross-modal consensus information of visual text to obtain the cross-modal encoding of visual text;

[0278] The target modality is one of the visual modality, text modality, and visual-text joint modality, and the visual features are image features or video features.

[0279] In this embodiment of the invention, the prior dictionary is initially a set of random variables. Through cross-modal learning between vision and text, prior knowledge is introduced into the prior dictionary to bridge the semantic misalignment between visual and textual content. Furthermore, to address the misalignment between the visual and textual modalities, a bridge is built between them. This bridge interacts with both the visual and textual modalities multiple times to select the most suitable target prior knowledge from the prior dictionary. Subsequent cross-modal encoding then aligns the visual and textual modalities.

[0280] In this embodiment of the invention, a prior knowledge dictionary is first constructed. Let the prior knowledge dictionary matrix be... Where C is the embedding length and D is the embedding dimension, the prior dictionary is constructed using random initialization.

[0281] Then, target prior knowledge is selected from the prior dictionary. After cross-modal learning of the visual and textual modalities, the prior dictionary learns relevant knowledge, which can be used as prior knowledge. This target prior knowledge is selected through visual content and textual information, and the selection result is called target prior knowledge. Since different visual emotion recognition tasks focus on different modalities, after determining the target modality based on the task objective of the visual emotion recognition task, target prior knowledge that matches the target modality is then selected from the prior dictionary based on the sample dataset.

[0282] When the target modality is a combined visual and textual modality, let the visual modality be... The text modality is ,in For the i-th modality in the visual modalities, For the j-th modality in the text modality, the best-fitting joint target needs to be selected from the C-dimensional prior dictionary U. K target prior knowledge items. When the target modality is a joint visual-text modality, based on the sample dataset, target prior knowledge items that fit the target modality are selected from the prior dictionary, which may include:

[0283] Visual-text modal pairs are constructed based on the sample dataset, and the first conditional mutual information between the prior knowledge in the prior dictionary and the visual-text modal pairs is calculated.

[0284] Establish a one-to-one mapping relationship between prior knowledge and first conditional mutual information to obtain a dictionary of first prior knowledge and conditional mutual information.

[0285] Based on the first prior knowledge-conditional mutual information dictionary values ​​in the first prior knowledge-conditional mutual information dictionary, in descending order, select the first preset number of prior knowledge that has the highest correlation with the visual text modality pair.

[0286] Select the second preset quantity of prior knowledge with the highest relevance from all visual text modal pairs as the target prior knowledge;

[0287] A visual-text modality pair includes a visual modality from the sample data and a text modality from the sample data.

[0288] The first conditional mutual information can be calculated using the following formula:

[0289] ;

[0290] in, The value of the first conditional mutual information. Let z be the joint probability distribution function of prior knowledge, text modality t, and visual modality v. Let z be the probability distribution function of prior knowledge z. Let be the joint probability distribution function of visual modality v and prior knowledge z. Let be the joint probability distribution function of the text modality t and the prior knowledge z. For the i-th modality in the visual modalities, Let Z be the j-th modality in the text modality, and let Z be the prior dictionary.

[0291] The first prior knowledge – the conditional mutual information dictionary value – can be calculated using the following formula:

[0292] ;

[0293] in, The first prior knowledge - conditional mutual information dictionary value. The value of the first conditional mutual information. For visual modality length, The length of the text modality.

[0294] Sort the dictionary values ​​of the first prior knowledge and conditional mutual information in descending order, and select the values ​​that are associated with the joint objective. The k most relevant prior knowledge items. To reduce the computational cost of subsequent encoding, after iterating through the above steps to select the k most relevant prior knowledge items for all combinations of visual and textual modalities, the prior knowledge items for all combinations are counted, sorted by frequency of occurrence, and the k most frequent prior knowledge items are selected as the final target prior knowledge items.

[0295] In other visual emotion recognition tasks, there is a need to place more emphasis on the visual modality or the textual modality, i.e., a joint objective. Become a single target The processing procedure is similar.

[0296] Because comments contain rich emotional and opinional information—direct reflections of viewers' feelings towards video content, such as emotional states and viewpoints—they can provide more direct and abundant clues for the task. In video sentiment recognition, sentiment and opinion analysis can be performed based on the content of both the video and the comments. In particular, by complementing different comments on the same video, video information can be obtained from multiple perspectives, ambiguities in the video content can be eliminated, and more emotional details can be captured. For example, different comments reflect diverse emotions and viewpoints of viewers. By analyzing these different comments, a more comprehensive emotional understanding can be obtained, thus acquiring multi-faceted video information from the same video. Sometimes the video content itself may be ambiguous or ambiguous; comments can provide additional contextual information to help interpret the video content and reduce misunderstandings. Details in comments can reveal subtle emotional details in the video, such as subtle humor, irony, or emotional shifts. Furthermore, by obtaining common value orientations in the comments—that is, by analyzing video comments—it helps to understand viewers' general attitudes towards a certain type of video, and also helps the model learn the values ​​and preferences of specific cultural or social groups. Therefore, for video sentiment recognition tasks based on comments, setting the target modality used to filter target prior knowledge to the text modality, and focusing on obtaining cross-modal consensus information from the text modality side, is more helpful for the model to understand the sentiment information of the video.

[0297] In some other possible implementations of the present invention, when the visual emotion recognition task to be processed is a comment-based visual emotion recognition task, the target modality is the text modality;

[0298] Based on the sample dataset, target prior knowledge that fits the target modality is selected from the prior dictionary, including:

[0299] Calculate the second conditional mutual information between prior knowledge in the prior dictionary and text modalities in the sample dataset;

[0300] Establish a one-to-one mapping relationship between prior knowledge and second conditional mutual information to obtain a dictionary of second prior knowledge-conditional mutual information.

[0301] Based on the second prior knowledge-conditional mutual information dictionary values ​​in the second prior knowledge-conditional mutual information dictionary, the third preset quantity of prior knowledge with the highest correlation to the text modalities in the sample dataset is selected in descending order.

[0302] The fourth preset quantity of prior knowledge, which has the highest correlation with each text modality in the sample dataset, is selected as the target prior knowledge.

[0303] When the target modality is a visual modality, the implementation method is similar to that when the target modality is a text modality.

[0304] Figure 4 This is a schematic diagram of a grounding encoder provided in an embodiment of the present invention.

[0305] Based on any of the methods for establishing cross-modal consensus information provided in the above embodiments, this invention further describes the implementation steps of visual text cross-modal encoding based on visual text cross-modal consensus information. It should be noted that the steps of visual text cross-modal encoding based on visual text cross-modal consensus information provided in this invention can be applied not only to the training process of visual emotion recognition models, but also to the training process of initial visual models to improve the model's cross-modal encoding capability, thereby enhancing the model's generalization ability.

[0306] In multimodal models, features from two modalities are typically directly input into a transformational encoder model. The self-attention mechanism of the transformational encoder model promotes the comparison and association of different modalities within the same representation space, thereby improving model performance. However, since semantic gaps often exist between the two modalities, forcibly associating the visual and textual modalities can weaken model performance. To improve cross-modal fusion, this invention retrieves relevant prior knowledge from a prior dictionary based on two or a single modality, using it as a bridge to enhance cross-modal fusion capabilities, such as... Figure 4 As shown, this embodiment of the invention provides a grounding encoder for implementing cross-modal information fusion of visual modalities and text modalities in a sample dataset based on prior knowledge of the target, and extracting visual-text cross-modal consensus information between the visual and text modalities.

[0307] In this embodiment of the invention, cross-modal information fusion is performed on the visual modal and text modal in the sample dataset based on prior knowledge of the target, and the visual-text cross-modal consensus information between the visual modal and the text modal is extracted, which may include:

[0308] Intramodal information integration and cross-modal information integration were performed on the visual modality and text modality in the sample dataset, respectively, to obtain the intramodal information integration result and the global consensus information integration result;

[0309] Based on the results of intramodal information integration and global consensus information integration, a text cross-attention module is constructed to search for visual information related to the text modality in the visual modality based on the prior knowledge of the target.

[0310] Based on the results of intramodal information integration and global consensus information integration, a visual cross-attention module is constructed to search for text information related to the visual modality in the text modality based on prior knowledge of the target.

[0311] Based on visual information associated with the text modality in the visual modality, textual information associated with the visual modality in the text modality, and prior knowledge of the target, an attention module is constructed to fuse and extract cross-modal consensus information between visual and text.

[0312] Specifically, intra-modal information integration and cross-modal information integration are performed on the visual modal and text modal in the sample dataset, respectively, to obtain intra-modal information integration results and global consensus information integration results, which may include:

[0313] Intramodal information is integrated from the visual modalities in the sample dataset to obtain visual modal information;

[0314] Intramodal information is integrated from the text modalities in the sample dataset to obtain text modal information;

[0315] Intramodal information integration of target prior knowledge yields target prior knowledge modal information;

[0316] Visual modal information, text modal information, and target prior knowledge modal information are used as the results of intramodal information integration;

[0317] Based on prior knowledge of the target, cross-modal information is integrated from visual and textual modalities to establish a cross-modal information association and interaction model of visual modal information, textual modal information, and target prior knowledge modal information;

[0318] Based on the cross-modal information association and interaction model, the transformation encoder model is invoked to integrate visual features, text features and target prior knowledge to obtain the global consensus information integration result;

[0319] Specifically, when processing visual features, masking is applied to text features and target prior knowledge; when processing text features, masking is applied to visual features and target prior knowledge; when processing target prior knowledge, no masking is applied to visual features, text features, or target prior knowledge.

[0320] like Figure 4The three boxes at the bottom center illustrate the intra-modal information integration of visual markers (visual modality), text markers (text modality), and prior knowledge markers (target prior knowledge), respectively, to integrate key and global information. Since some key information may be distributed in different locations, intra-modal information integration can capture this key information; furthermore, the self-attention mechanism itself can capture global semantic information.

[0321] Then as Figure 4 The intersection of the three boxes below illustrates the integration of all modal information—visual tags, text tags, and prior knowledge tags—based on prior knowledge tags. Since the prior knowledge tags are selected based on visual and text features, they are connected not only to themselves but also to visual and text tags, thereby enhancing the correlation and interaction between different modalities.

[0322] like Figure 4 As shown, the same transformation encoder model is used to process visual tags, text tags, and prior knowledge tags. Here, the transformation encoder model can be a multi-head attention model, a Bidirectional Encoder Representations from Transformers (BERT) model based on the transformation encoder model, or a Contrastive Language-Image Pretraining (CLIP) model. The model is denoted as... Make the visual features Text features are The features of the selected prior knowledge are The corresponding token type is ,in , , These are the token types corresponding to visual tags, text tags, and prior knowledge tags. ,in , , The input is a mask corresponding to visual tags, text tags, and prior knowledge tags. It includes the following three mask types:

[0323] A visual feature mask is used to obscure the other two modalities when processing visual features. =1, , If it is 0, it is denoted as ;

[0324] A text feature mask is used to obscure the other two modalities when processing text features. =1, , If it is 0, it is denoted as ;

[0325] A mask that integrates all modal information based on prior knowledge of the target is used when processing prior knowledge of the target, since it is necessary to consider all modal information, i.e., let , , It is 1, denoted as ;

[0326] By combining the above three masks, we obtain = Through transformational encoder model processing, a cross-modal information association and interaction model is obtained, which integrates visual modal information, textual modal information, and target prior knowledge modal information. The result of cross-modal information integration is... .

[0327] Following the above information integration steps, key information and global common information have been integrated. To further improve consensus information, based on the target prior knowledge as a bridge, the visual modality and text modality interact with each other to establish relationships between visual and linguistic information. This ensures that each modality considers the context of the other modality, improving the model's understanding of information within each modality. Based on the results of intra-modal information integration and global consensus information integration, a text cross-attention module is constructed to search for visual information in the visual modality that is associated with the text modality based on the target prior knowledge. This module may include:

[0328] The visual modal information and target prior knowledge in the merged intramodal information integration result are used as key-value data for the text cross-attention module. The text modal information in the merged intramodal information integration result is used as the query statement for the text cross-attention module. The text cross-attention module is used to query the corresponding key-value data based on the query statement to obtain the visual information in the visual modality associated with the text modality.

[0329] Based on the results of intra-modal information integration and global consensus information integration, a visual cross-attention module is constructed to search for textual information in the textual modality that is associated with the visual modality based on prior knowledge of the target, including:

[0330] The text modal information and target prior knowledge in the merged intramodal information integration result are used as key-value data for the visual cross-attention module. The visual modal information in the merged intramodal information integration result is used as the query statement for the visual cross-attention module. The visual cross-attention module is then used to query the corresponding key-value data based on the query statement to obtain the text information in the text modality that is associated with the visual modality.

[0331] like Figure 4 As shown, a text cross-attention module is constructed to acquire the visual information corresponding to the text, which helps the model understand the specific content in an image or video. By constructing the query, key, and value in the text cross-attention module, visual modality and prior knowledge of the target are merged and used as the key and value, respectively. ,in and They are visual modalities and target prior knowledge, respectively, with text modalities as... Using the text modality as the query and the visual modality and prior knowledge as the key and value, we retrieve visual information related to the text modality. A multi-head attention mechanism can be used as the text cross-attention module here. The output dimension is the same as the query dimension, and the attention header is 3.

[0332] like Figure 4 As shown, a vision cross-attention module is constructed to acquire the corresponding textual information, which helps the model understand the specific context of the description or text. The vision cross-attention module is constructed using a query, a key, and a value. Here, the visual modality is used as the query, and the text modality and target prior knowledge are used as the key and value. That is, the text modality and target prior knowledge are combined. ,in and These are textual modalities and target prior knowledge, respectively, with the visual modal as... To acquire textual information related to the visual modality; a multi-head attention mechanism can be used as the visual cross-attention module here. The output dimension is the same as the query dimension, and the attention header is 3.

[0333] While the above steps acquire information from other modalities at different levels, this information is discretized and needs to be fused and refined to enable the model to better handle and understand complex tasks that simultaneously involve two modalities. Therefore, based on visual information associated with the text modality in the visual modality, textual information associated with the visual modality in the text modality, and prior knowledge of the target, an attention module is constructed to fuse and refine cross-modal consensus information between visual and text. This can include:

[0334] Visual information in the visual modality that is associated with the text modality is used as text tags, text information in the text modality that is associated with the visual modality is used as visual tags, and prior knowledge of the target is used as prior knowledge tags.

[0335] After splicing visual tags, prior knowledge tags, and text tags, the input is fed into the attention module to fuse and extract cross-modal consensus information between visual and textual data.

[0336] In the attention module, visual tags and prior knowledge tags are merged into key-value data, with text tags as the query statement, and text tags and prior knowledge tags are merged into key-value data, with visual tags as the query statement, and visual-text cross-modal consensus information is output.

[0337] like Figure 4 As shown, visual markers, prior knowledge markers, and text markers are combined, that is... ,in Output results for the visual cross-attention module. Output the results for the text cross-attention module. For the model The processed prior knowledge is labeled. Consensus information is further integrated and refined based on the attention mechanism. Here, a multi-head attention mechanism can still be used as the attention module, where the query, key, and value are the same, and the input and output dimensions are identical. .

[0338] Figure 5 This is a flowchart of a video frame-text cross-modal coding method provided in an embodiment of the present invention.

[0339] Because everyone has different focuses, the same image or video frame may have different text content (i.e., comments). By constructing the relationship between the visual features of video frames and the text features, we can solve the problem of text focusing on a certain frame or some frames in the video. However, due to the semantic inconsistency between the two, in order to alleviate this phenomenon, we can filter the target prior knowledge with similar prior dictionaries based on the text semantics and the visual semantics of video frames to serve as cross-modal consensus information between video frames (images) and text and realize cross-modal consensus encoding.

[0340] like Figure 5 As shown, when the visual features are image features (or video frame features), cross-modal fusion encoding is performed based on the visual features, text features, and target prior knowledge corresponding to the visual-text cross-modal consensus information to obtain visual-text cross-modal encoding, which may include:

[0341] The output results of multiple executions of target prior knowledge adapted to the target modality by selecting from the prior dictionary and extracting visual-text cross-modal consensus information between visual and text modalities are encoded to obtain visual-text cross-modal coding.

[0342] The output of the current iteration serves as the input data for the next iteration, which involves selecting target prior knowledge from the prior dictionary that is compatible with the target modality and extracting visual-text cross-modal consensus information between the visual and text modalities.

[0343] like Figure 5 As shown, in this embodiment of the invention, referring to the description of the above embodiments of the invention, after the input video data is split into video frames, the visual features of the frames are extracted by the video frame feature extraction module. (0, 1, ..., N).

[0344] Obtain text samples corresponding to video samples, and extract text features using a language backbone model. (0, 1, ..., M), where Language backbone models include models such as Bidirectional Encoder Representations from Transformers (BERT) and RoBERTa, which are based on transformational encoder models. For example, if the text sample is an evaluation of "how beautiful the last shot is, sliding sideways, what amazing technique", the sentence "[classification tag][how]...[amazing][sequence termination tag][complete character]" is extracted, and the language backbone network is used to extract the text features in the sentence.

[0345] Initialize a prior dictionary (1, 2, ..., C), combine frame visual features with text features, and use the visual and text modal features from the prior dictionary to select the K most similar prior knowledge points, denoted as C. ,in For visual modality The i-th frame mode, For text model The j-th mode.

[0346] Visual features of spliced ​​frames Text features K prior knowledge points for the target, denoted as Encoding is performed using the grounding encoder module provided in the above embodiments of the present invention to obtain... .

[0347] Obtain the features of the encoded video frames Text features The data is input into the prior dictionary module, and then the K most similar prior knowledge items in the prior dictionary are selected again, denoted as... .

[0348] The output results of multiple executions, including filtering target prior knowledge from a priori dictionary to match the target modality and extracting cross-modal consensus information between the visual and textual modalities, are fused using a residual structure and then encoded to obtain cross-modal visual-text coding. This coding can include:

[0349] use Perform feature fusion processing on the image feature encoding in the i-th output result to obtain the image feature fusion result corresponding to the i-th output result;

[0350] use Perform feature fusion processing on the text feature encoding in the i-th output result to obtain the text feature fusion result corresponding to the i-th output result;

[0351] use The prior knowledge encoding in the i-th output result is fused to obtain the prior knowledge fusion result corresponding to the i-th output result;

[0352] By concatenating the image feature fusion result corresponding to the i-th output result, the text feature fusion result corresponding to the i-th output result, and the prior knowledge fusion result corresponding to the i-th output result, the feature fusion result corresponding to the i-th output result is obtained.

[0353] If i is not N, then for the feature fusion result corresponding to the i-th output result, the target prior knowledge that is adapted to the target modality is selected from the prior dictionary and the visual-text cross-modal consensus information between the visual modality and the text modality is extracted to obtain the i+1-th output result.

[0354] If i is N, then the feature fusion result corresponding to the i-th output result is used as the visual text cross-modal encoding;

[0355] in, This represents the image feature fusion result corresponding to the i-th output. The residual coefficient is... Encode the image features in the i-th output result. For the input image features, This represents the text feature fusion result corresponding to the i-th output. Encode the text features in the i-th output result. The features of the input text. This represents the prior knowledge fusion result corresponding to the i-th output result. Encode the prior knowledge in the i-th output result. The input is the target prior knowledge.

[0356] That is to say, the grounding encoder provided in the embodiments of the present invention is used to repeatedly execute the steps of filtering target prior knowledge based on video frame features (or image features) and text features and performing visual text cross-modal encoding, with the output result of the previous grounding encoder as the input data of the current grounding encoder, and so on for N times.

[0357] For example, using residual structure fusion get , and After splicing The input is then encoded again in the grounding encoder module to obtain... Repeat this step N times to obtain the final encoded result. .

[0358] It should be noted that the video frame-text cross-modal coding method provided in this embodiment of the invention can be applied to the training process of the visual emotion recognition model provided in this embodiment of the invention, and can also be applied to the training process of the initial visual model provided in this embodiment of the invention.

[0359] The visual emotion recognition method provided in this invention also solves the problem of text modality focusing on global and local information of an image by cross-modal coding at the image level based on residual structure. This helps the model capture detailed information of the image, thereby enhancing the model's understanding of the semantic correspondence between the visual and text modalities and improving its performance in related tasks such as image language understanding.

[0360] Figure 6 This is a flowchart of a video-text cross-modal coding method provided in an embodiment of the present invention.

[0361] Video frame-to-text cross-modal coding addresses the issue of comments focusing on a single frame or several frames of a video. However, for comments that focus on a segment or several segments, or even the entire video, frame-level processing becomes insufficient. To solve this problem, it's necessary to establish the relationship between video features and text. Similarly, to address the semantic inconsistency between the two, target prior knowledge in a prior dictionary is filtered based on both textual and video semantics. For example... Figure 6 As shown, when the visual features are video features, cross-modal fusion encoding is performed based on the visual features, text features, and target prior knowledge corresponding to the visual-text cross-modal consensus information to obtain visual-text cross-modal encoding, which may include:

[0362] The output results of multiple executions of target prior knowledge adapted to the target modality by selecting from the prior dictionary and extracting visual-text cross-modal consensus information between visual and text modalities are fused using learnable residual parameters and then encoded to obtain visual-text cross-modal coding.

[0363] The output of the current iteration serves as the input data for the next iteration, which involves selecting target prior knowledge from the prior dictionary that is compatible with the target modality and extracting visual-text cross-modal consensus information between the visual and text modalities.

[0364] like Figure 6 As shown, in this embodiment of the invention, referring to the description of the above embodiments of the invention, after the input video data is split into video frames, the visual features of the frames are extracted by the video frame feature extraction module. (0, 1, ..., N).

[0365] Obtain the subject and object from the hashtags and comment text (evaluations) corresponding to the video samples, and extract text features using a language backbone model. (0, 1, ..., M). The subject and object are obtained through syntactic analysis and dependency relations.

[0366] The input video frame features and text features (topic tags, subject, and object) are encoded using a hashtag-based cross encoder, and then the video features are obtained through a multilayer perceptron. .

[0367] The text features of the evaluation text corresponding to the video are extracted using a language backbone model. ,Right now .

[0368] For example, the annotation text is "Yao and McGrady performed the best in the basketball team." Using the search keyword "basketball" as the hashtag, syntactic analysis yields the sentence "[Classification tag][Basketball][Yao][McGrady][Sequence termination tag][Complete character]", which, after concatenation of the hashtag, subject, and object. Simultaneously, the annotation text is converted into the sentence "[Classification tag][Yao]……[Team][Sequence termination tag][Complete character]". After extracting features from "[Classification tag][Basketball][Yao][McGrady][Sequence termination tag][Complete character]", the hashtag-based cross-encoder provided in this embodiment extracts video features based on the extracted text features from "[Classification tag][Basketball][Yao][McGrady][Sequence termination tag][Complete character]", and outputs the video features through a multilayer perceptron. Finally, the hashtag "[Classification tag][Yao]……[Team][Sequence termination tag][Complete character]" is used to extract text features from the hashtag.

[0369] Based on video features and text features To achieve the joint objective, the K closest prior tokens are selected from the frame prior dictionary set. Since a video consists of frames, and each frame reflects video-related information to some extent, the selected target prior knowledge from all frames is used as a candidate set for the video's prior knowledge dictionary; this candidate set is a subset of the prior dictionary. Specifically:

[0370] Obtain the top-k prior knowledge for all frame filtering, denoted as set A, and , , Prior knowledge of the target to be selected for the i-th frame. ,in The top-k target prior knowledge is used as input to the i-th grounding encoder for filtering.

[0371] The K closest target priors from A are selected based on the size of the prior knowledge-conditional mutual information dictionary values, denoted as KK. ,in For visual modality The i-th mode, For text modality The j-th mode.

[0372] Features of spliced ​​videos Text features K prior knowledge points for the target, denoted as The encoding is performed through a grounding encoder module to obtain the code. .

[0373] Obtain the encoded video features respectively Text features They are combined into a joint objective, and from the prior token set A selected from all frames, the K most similar prior knowledge of the objective are selected again, denoted as... .

[0374] The visual-text cross-modal coding is obtained by fusing the output results of multiple executions of target prior knowledge adapted to the target modality from a self-prior dictionary and extracting visual-text cross-modal consensus information between the visual and text modalities using learnable residual parameters. This encoding can include:

[0375] For the video features in the i-th output result, set the corresponding frame coefficients for each video frame, obtain the video residual block features based on the frame coefficients and the frame features of the video frames, and fuse the video residual block features with the video features in the i-th output result to obtain the video feature fusion result corresponding to the i-th output result.

[0376] For the text features in the i-th output result, set the corresponding text tag coefficient for each text tag, obtain the text residual block features based on the text tag coefficient and text tag, and fuse the text residual block features with the text features in the i-th output result to obtain the text feature fusion result corresponding to the i-th output result;

[0377] For the target prior knowledge in the i-th output result, set the corresponding prior knowledge label coefficient for each prior knowledge label. Obtain the prior knowledge residual block feature based on the prior knowledge label coefficient and the prior knowledge label. Fuse the prior knowledge residual block feature with the target prior knowledge in the i-th output result to obtain the prior knowledge fusion result corresponding to the i-th output result.

[0378] By concatenating the video feature fusion result corresponding to the i-th output result, the text feature fusion result corresponding to the i-th output result, and the prior knowledge fusion result corresponding to the i-th output result, the feature fusion result corresponding to the i-th output result is obtained.

[0379] If i is not N, then for the feature fusion result corresponding to the i-th output result, the target prior knowledge that is adapted to the target modality is selected from the prior dictionary and the visual-text cross-modal consensus information between the visual modality and the text modality is extracted to obtain the i+1-th output result.

[0380] If i is N, then the feature fusion result corresponding to the i-th output result is used as the visual text cross-modal encoding.

[0381] like Figure 6As shown, video features, text features, and target prior knowledge features are fused using learnable residual coefficients. Specifically, each frame is used to learn corresponding coefficients, and residual block features are obtained from the frame coefficients and frame features, which are then fused using a residual structure. Let the features of the video frame be F, with dimensions N×w×h, where N is the number of video frames, and w and h are the width and height, respectively (here, they can be 192 and 768). The above video feature fusion can include:

[0382] use Calculate the global average value in the frame count dimension;

[0383] Construct a gating mechanism to calculate the residual coefficients for each frame. ,in For a fully connected network, For activation functions, such as the ReLU (Rectified Linear Unit) activation function in neural networks. This is the sigmoid function;

[0384] Calculate the video features of the residual block. ;

[0385] Video features are fused using residual structures. ;

[0386] Text feature fusion and target prior knowledge feature fusion learn corresponding coefficients based on the number of tokens, and then fuse them using the same method as video feature fusion to obtain... and .

[0387] The feature fusion result is obtained by combining the video feature fusion result, the text feature fusion result, and the prior knowledge fusion result. .

[0388] Repeat the above steps N times to obtain the final encoding result. .

[0389] That is to say, the grounding encoder provided in the embodiments of the present invention is used to repeatedly execute the steps of filtering target prior knowledge based on video features and text features and performing visual text cross-modal encoding, with the output result of the previous grounding encoder as the input data of the current grounding encoder, and so on for N times.

[0390] It should be noted that the video-text cross-modal coding method provided in this embodiment of the invention can be applied to the training process of the visual emotion recognition model provided in this embodiment of the invention, and can also be applied to the training process of the initial visual model provided in this embodiment of the invention.

[0391] The visual emotion recognition method provided in this invention also performs cross-modal coding at the video level based on learnable residual coefficients. By learning the residual coefficients of different video frames, it helps the model handle more complex video language understanding tasks. At the same time, it also deeply constructs the relationship between the visual and language modalities, improving the model generalization ability of the visual emotion recognition model.

[0392] It should be noted that in the embodiments of the visual emotion recognition methods of the present invention, some steps or features may be ignored or not executed. The hardware or software functional modules described are for ease of explanation and are not the only implementation of the visual emotion recognition methods provided in the embodiments of the present invention.

[0393] The various embodiments of the visual emotion recognition method have been described in detail above. Based on this, the present invention also discloses a visual emotion recognition device, equipment and readable storage medium corresponding to the above method.

[0394] Figure 7 This is a schematic diagram of the structure of a visual emotion recognition device provided in an embodiment of the present invention.

[0395] like Figure 7 As shown, the visual emotion recognition device provided in this embodiment of the invention includes:

[0396] The first acquisition unit 701 is used to acquire the initial visual model and sample dataset;

[0397] The first training unit 702 is used to configure a first attention head based on text sentiment evaluation coding, a second attention head based on visual text cross-modal consensus information, and a third attention head that integrates text sentiment evaluation coding and visual text cross-modal coding for the initial visual model. After concatenating the loss values ​​of each attention head into the model loss value of the initial visual model, the weights of each attention head in the initial visual model are trained using the sample dataset, the task objective of the visual emotion recognition task to be processed, and the loss values ​​of each attention head, and the visual emotion recognition model is output.

[0398] The first computing unit 703 is used to respond to the visual emotion recognition task to be processed by calling the visual emotion recognition model to perform emotion recognition processing on the input data to be recognized, and to obtain the visual emotion recognition result.

[0399] Specifically, based on the task objective of the visual emotion recognition task to be processed, the sample dataset includes image data or video data with annotated text; visual-text cross-modal consensus information is constructed through mutual information between the visual modality and the text modality.

[0400] It should be noted that in the various embodiments of the visual emotion recognition device provided in this invention, the division of units is only a logical functional division, and other division methods can be used. The connection between different units can be electrical, mechanical, or other connection methods. Separate units can be located in the same physical location or distributed across multiple network nodes. Each unit can be implemented in hardware or as a software functional unit. That is, some or all of the units provided in this invention can be selected according to actual needs, and corresponding connection or integration methods can be used to achieve the purpose of the solution in this invention.

[0401] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0402] Figure 8 This is a schematic diagram of the structure of a visual emotion recognition device provided in an embodiment of the present invention.

[0403] like Figure 8 As shown, the visual emotion recognition device provided in this embodiment of the invention includes:

[0404] Memory 810 is used to store computer program 811;

[0405] Processor 820 is configured to execute computer program 811, which, when executed by processor 820, implements the steps of the visual emotion recognition method as described in any of the above embodiments.

[0406] The processor 820 may include one or more processing cores, such as a 3-core processor or an 8-core processor. The processor 820 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 820 may also include a main processor and a coprocessor. The main processor, also known as a Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 820 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 820 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0407] The memory 810 may include one or more readable storage media, which may be non-transitory. The memory 810 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 810 is used to store at least the following computer program 811, wherein, after being loaded and executed by the processor 820, the computer program 811 is able to implement the relevant steps in the visual emotion recognition method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 810 may also include an operating system 812 and data 813, and the storage method may be temporary storage or permanent storage. The operating system 812 may be Windows, Linux, or other types of operating systems. The data 813 may include, but is not limited to, the data involved in the above methods.

[0408] In some embodiments, the visual emotion recognition device may further include a display screen 830, a power supply 840, a communication interface 850, an input / output interface 860, a sensor 870, and a communication bus 880.

[0409] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on visual emotion recognition devices and may include more or fewer components than illustrated.

[0410] The visual emotion recognition device provided in this embodiment of the invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the visual emotion recognition method as described above, with the same effect.

[0411] This invention provides a readable storage medium storing a computer program thereon, which, when executed by a processor, can implement the steps of the visual emotion recognition method as described in any of the above embodiments.

[0412] The readable storage medium may include: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program code.

[0413] For a description of the readable storage medium provided in the embodiments of the present invention, please refer to the above method embodiments, and its effect is the same as that of the visual emotion recognition method provided in the embodiments of the present invention. The present invention will not repeat the details here.

[0414] The foregoing has provided a detailed description of a visual emotion recognition method, apparatus, device, and readable storage medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus, device, and readable storage medium disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

[0415] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A visual emotion recognition method, characterized in that, include: Obtain the initial visual model and sample dataset; Configure the initial visual model with a first attention head based on text sentiment evaluation coding, a second attention head based on visual text cross-modal consensus information, and a third attention head that fuses the text sentiment evaluation coding and the visual text cross-modal coding. After concatenating the loss values ​​of each attention head into the model loss value of the initial visual model, use the sample dataset, the task objective of the visual sentiment recognition task to be processed, and the loss values ​​of each attention head to train the weights of each attention head in the initial visual model, and output the visual sentiment recognition model. In response to the visual emotion recognition task to be processed, the visual emotion recognition model is invoked to perform emotion recognition processing on the input data to be recognized, and the visual emotion recognition result is obtained. Wherein, according to the task objective of the visual emotion recognition task to be processed, the sample dataset includes image data or video data with annotated text; the visual text cross-modal consensus information is constructed through mutual information between the visual modality and the text modality; The visual text cross-modal encoding based on the visual text cross-modal consensus information is obtained through the following steps: Initialize the prior dictionary; Determine the target modality based on the task objective of the visual emotion recognition task to be processed; Based on the sample dataset, target prior knowledge that is compatible with the target modality is selected from the prior dictionary; Based on the prior knowledge of the target, cross-modal information fusion is performed on the visual modality and the text modality in the sample dataset to extract the visual-text cross-modal consensus information between the visual modality and the text modality; Cross-modal fusion encoding is performed based on the visual features, text features, and target prior knowledge corresponding to the cross-modal consensus information of the visual text to obtain the cross-modal encoding of the visual text; The target modality is one of a visual modality, a text modality, and a combined visual-text modality, and the visual feature is an image feature or a video feature; The process of fusing cross-modal information between the visual modalities and text modalities in the sample dataset based on the target prior knowledge, and extracting the visual-text cross-modal consensus information between the visual and text modalities, includes: Intramodal information integration and cross-modal information integration are performed on the visual modal and text modal in the sample dataset, respectively, to obtain the intramodal information integration result and the global consensus information integration result; Based on the intramodal information integration results and the global consensus information integration results, a text cross-attention module is constructed to search for visual information related to the text modality in the visual modality based on the target prior knowledge; Based on the intramodal information integration results and the global consensus information integration results, a visual cross-attention module is constructed to search for text information related to the visual modality in the text modality based on the target prior knowledge; Based on visual information associated with the text modality in the visual modality, textual information associated with the visual modality in the text modality, and the prior knowledge of the target, an attention module is constructed to fuse and extract the cross-modal consensus information of the visual and textual data.

2. The visual emotion recognition method according to claim 1, characterized in that, The visual emotion recognition task to be processed is a video emotion recognition task; The second attention head includes a fourth attention head based on video frame text cross-modal consensus information and a fifth attention head based on video text cross-modal consensus information.

3. The visual emotion recognition method according to claim 2, characterized in that, The model loss value is expressed by the following formula: ； in, The model parameters for the i-th attention head are... The weight of the i-th attention head, Let i be the loss value of the attention head. Let i be the model loss value; i = 0, 1, 2, 3; The attention head includes the first attention head, the fourth attention head, the fifth attention head, and the third attention head.

4. The visual emotion recognition method according to claim 3, characterized in that, The loss value of the attention head is expressed by the following formula: ； The output of the visual emotion recognition model is expressed by the following formula: ； in, , , , ; in, For sentiment classification loss weights, Weighting loss for opinion categories For sentiment classification loss value, Loss value for opinion classification, Here, L is the loss weight calculation function, and L is the concatenated value of the sentiment classification loss value and the opinion classification loss value. It is a fully connected layer. For activation function, For regression function, For the sentiment classification results, For the results of opinion classification, The video emotion recognition model outputs the model calculation results based on the input video data and text data.

5. The visual emotion recognition method according to claim 1, characterized in that, The visual emotion recognition task to be processed is a video emotion recognition task; The process of training the weights of each attention head in the initial visual model using the sample dataset, the task objective of the visual emotion recognition task to be processed, and the loss values ​​of each attention head, and outputting the visual emotion recognition model, includes: For video samples in the sample dataset, multiple video frames are extracted from the video samples, the frame visual features of the video frames are obtained, and time dimension information is added to the frame visual features according to the order of the video frames in the video samples. Extract text features based on the annotation text corresponding to the video sample; After encoding the frame visual features and the text features with added time dimension information, the video features of the video sample are obtained; The video features are input into the initial visual model, and the weights of each attention head in the initial visual model are adjusted according to the loss value of each attention head, and a visual emotion recognition model is output.

6. The visual emotion recognition method according to claim 5, characterized in that, The step of extracting text features based on the annotation text corresponding to the video sample includes: Extract the subject and object from the annotation text corresponding to the video sample; The text features are obtained by concatenating the subject and object with the search keywords used to obtain the video samples.

7. The visual emotion recognition method according to claim 5, characterized in that, The step of extracting text features based on the annotation text corresponding to the video sample includes: If the video sample corresponds to video segmentation information and segmentation event description, then the video sample is divided into multiple sub-video samples according to the video segmentation information, and the corresponding segmentation event description is used as the annotation text of the sub-video sample; The video action recognition model is invoked to identify and predict the action of the sub-video sample. After extracting video description keywords from the annotation text of the sub-video sample, the action prediction result of the sub-video sample is concatenated with the video description keywords of the sub-video sample to obtain the text features of the sub-video sample; The text features of each of the sub-video samples are integrated into the text features of the video sample.

8. The visual emotion recognition method according to claim 5, characterized in that, The step of extracting text features based on the annotation text corresponding to the video sample includes: If the video sample corresponds to video segmentation information and segmentation action description, then the video sample is divided into multiple sub-video samples according to the video segmentation information; The video description model is invoked to generate description text for each of the sub-video samples; After extracting action keywords from the segmented action description and video description keywords from the description text, the action keywords and video description keywords of the sub-video sample are concatenated to obtain the text features of the sub-video sample. The text features of each of the sub-video samples are integrated into the text features of the video sample.

9. The visual emotion recognition method according to claim 1, characterized in that, The initial visual model is obtained through the following steps: The initial visual model is obtained by constructing a loss function based on visual text matching task, masked text prediction task and consensus information filtering task; The visual-text matching task is used to identify the matching status between visual samples and text samples; the masked text prediction task is used to identify the information to be masked based on the context information that is not masked; and the consensus information filtering task is used to filter the consensus information of visual modalities and text modalities from the prior dictionary.

10. The visual emotion recognition method according to claim 9, characterized in that, The loss function corresponding to the visual text matching task is: ； in, Let S be the loss value for the visual-text matching task, and S be the number of matching pairs between the visual modality and the text modality. For sign functions, when the visual modality matches the text modality A value of 1 indicates a mismatch between the visual and textual modalities. =0, For the i-th visual modality, For the i-th text modality, The probability of matching visual modalities with text modalities.

11. The visual emotion recognition method according to claim 9, characterized in that, The loss function corresponding to the masked text prediction task is: ； ； in, Let S be the first mask text prediction loss value, S be the number of matching pairs between the visual modality and the text modality, and V be the dictionary size of the text modality. Let be the sign function, which is defined when the visual label predicted based on the v-th visual modality matches the masked visual label in the i-th text modality. The value is 1 when the visual label predicted based on the v-th visual modality does not match the masked visual label in the i-th text modality. =0, For the i-th visual modality, For the i-th text modality, To predict the probability of a masked visual label based on the input visual and textual modalities; The loss value for predicting the second masked text. Let be the sign function, which is defined when the annotation text predicted based on the v-th visual modality matches the masked annotation text in the i-th text modality. The value is 1 when the annotation text predicted based on the v-th visual modality does not match the masked annotation text in the i-th text modality. =0, Let i be the visual label for the i-th visual modality. Let be the masked annotation text in the i-th text modality. To predict the probability of a masked text modality based on the input visual modality, visual label, and remaining text modality.

12. The visual emotion recognition method according to claim 9, characterized in that, The loss function corresponding to the consensus information filtering task is: ； in, The loss function corresponding to the consensus information filtering task is defined, where K is the amount of prior knowledge of the target. For vectorized functions, For weight parameters, For the i-th visual feature, For the j-th text feature, To associate relevant information between the i-th visual modality and the j-th text modality, Let be the dictionary value of conditional mutual information between the i-th visual modality, the j-th text modality, and the relevant information relating the i-th visual modality and the j-th text modality. For information between visual and textual modalities, Let be the conditional mutual information dictionary values ​​for the i-th visual modality, the j-th text modality, and the relevant information between the visual and text modalities. The dictionary value of conditional mutual information between all visual modalities, the j-th text modal, and the relevant information relating the i-th visual modal and the j-th text modal. is the vector for each feature in the prior dictionary.

13. The visual emotion recognition method according to claim 1, characterized in that, When the visual emotion recognition task to be processed is a comment-based visual emotion recognition task, the target modality is the text modality; The step of selecting target prior knowledge that matches the target modality from the prior dictionary based on the sample dataset includes: Calculate the second conditional mutual information between the prior knowledge in the prior dictionary and the text modalities in the sample dataset; Establish a one-to-one mapping relationship between the prior knowledge and the second conditional mutual information to obtain the second prior knowledge-conditional mutual information dictionary; Based on the second prior knowledge-conditional mutual information dictionary values ​​in the second prior knowledge-conditional mutual information dictionary in descending order, select the third preset number of prior knowledge that has the highest correlation with the text modalities in the sample dataset; The target prior knowledge is selected from the third preset number of prior knowledge that has the highest correlation with each text modality in the sample dataset, and the fourth preset number of prior knowledge that has the highest correlation is selected as the target prior knowledge.

14. The visual emotion recognition method according to claim 1, characterized in that, The process involves integrating intra-modal information and cross-modal information on the visual modal and text modal of the sample dataset, respectively, to obtain intra-modal information integration results and global consensus information integration results, including: Intramodal information is integrated from the visual modalities in the sample dataset to obtain visual modal information; Intramodal information is integrated from the text modalities in the sample dataset to obtain text modal information; Intramodal information integration is performed on the aforementioned prior knowledge of the target to obtain modal information of the prior knowledge of the target; The visual modal information, the text modal information, and the target prior knowledge modal information are used as the result of the intramodal information integration; Based on the target prior knowledge, cross-modal information is integrated between visual modality and text modality to establish a cross-modal information association and interaction model of the visual modality information, the text modality information, and the target prior knowledge modality information; Based on the cross-modal information association and interaction model, the transformation encoder model is invoked to integrate visual features, text features, and the target prior knowledge to obtain the global consensus information integration result. Specifically, when processing the visual features, the text features and the target prior knowledge are masked; when processing the text features, the visual features and the target prior knowledge are masked; when processing the target prior knowledge, no masks are set for the visual features, the text features, and the target prior knowledge.

15. The visual emotion recognition method according to claim 1, characterized in that, The step of constructing a text cross-attention module based on the intra-modal information integration result and the global consensus information integration result to search for visual information in the visual modality associated with the text modality based on the target prior knowledge includes: The visual modal information and the target prior knowledge in the intramodal information integration result are merged into the key-value data of the text cross-attention module. The text modal information in the intramodal information integration result is used as the query statement of the text cross-attention module. The text cross-attention module is used to query the corresponding key-value data based on the query statement to obtain the visual information in the visual modality associated with the text modality. The step of constructing a visual cross-attention module based on the intra-modal information integration result and the global consensus information integration result to search for text information in the text modality associated with the visual modality based on the target prior knowledge includes: The text modal information and the target prior knowledge in the intramodal information integration result are merged into the key-value data of the visual cross-attention module. The visual modal information in the intramodal information integration result is used as the query statement of the visual cross-attention module. The visual cross-attention module is used to query the corresponding key-value data based on the query statement to obtain the text information in the text modality associated with the visual modality.

16. The visual emotion recognition method according to claim 1, characterized in that, The aforementioned attention module, constructed based on visual information associated with the textual modality in the visual modality, textual information associated with the visual modality in the textual modality, and the target prior knowledge, is used to fuse and extract the cross-modal consensus information between visual and textual data. Visual information in the visual modality that is associated with the text modality is used as text tags, text information in the text modality that is associated with the visual modality is used as visual tags, and the target prior knowledge is used as prior knowledge tags. After splicing the visual markers, the prior knowledge markers, and the text markers, the information is input into the attention module to fuse and extract the cross-modal consensus information of the visual text. In the attention module, the visual tags and the prior knowledge tags are merged into key-value data, and the text tags are used as query statements. Additionally, the text tags and the prior knowledge tags are merged into key-value data, and the visual tags are used as query statements to output the visual-text cross-modal consensus information.

17. The visual emotion recognition method according to claim 1, characterized in that, The visual features are image features; The step of performing cross-modal fusion encoding based on the visual features, text features, and target prior knowledge corresponding to the cross-modal consensus information of the visual text to obtain the cross-modal encoding of the visual text includes: The output results of multiple executions of the target prior knowledge adapted to the target modality by selecting from the prior dictionary and the visual-text cross-modal consensus information between the visual modality and the text modality are fused using a residual structure and then encoded to obtain the visual-text cross-modal encoding. The output of the current iteration serves as the input data for the next iteration, which involves selecting target prior knowledge from the prior dictionary that is compatible with the target modality and extracting visual-text cross-modal consensus information between the visual and text modalities.

18. The visual emotion recognition method according to claim 17, characterized in that, The process of fusing the output results after multiple executions of the process of selecting target prior knowledge from the prior dictionary that is compatible with the target modality and extracting the visual-text cross-modal consensus information between the visual and text modalities using a residual structure, and then encoding the results, yields the visual-text cross-modal encoding, including: use The image feature encoding in the i-th output result is subjected to feature fusion processing to obtain the image feature fusion result corresponding to the i-th output result; use The text feature encoding in the i-th output result is subjected to feature fusion processing to obtain the text feature fusion result corresponding to the i-th output result; use The prior knowledge encoding in the i-th output result is fused to obtain the prior knowledge fusion result corresponding to the i-th output result; By concatenating the image feature fusion result corresponding to the i-th output result, the text feature fusion result corresponding to the i-th output result, and the prior knowledge fusion result corresponding to the i-th output result, the feature fusion result corresponding to the i-th output result is obtained; If i is not N, then for the feature fusion result corresponding to the i-th output result, the target prior knowledge adapted to the target modality is selected from the prior dictionary and the visual-text cross-modal consensus information between the visual modality and the text modality is extracted to obtain the (i+1)-th output result; If i is N, then the feature fusion result corresponding to the i-th output result is used as the cross-modal encoding of the visual text; in, The image feature fusion result corresponding to the output result of the i-th iteration is... For residual coefficients, Encode the image features in the output result of the i-th iteration. For the input image features, This represents the text feature fusion result corresponding to the output result of the i-th iteration. Encode the text features in the output result of the i-th iteration. The features of the input text. The prior knowledge fusion result corresponding to the output result of the i-th iteration. Encode the prior knowledge in the output result of the i-th iteration. The input is the target prior knowledge.

19. The visual emotion recognition method according to claim 1, characterized in that, The visual features are video features; The step of performing cross-modal fusion encoding based on the visual features, text features, and target prior knowledge corresponding to the cross-modal consensus information of the visual text to obtain the cross-modal encoding of the visual text includes: The visual-text cross-modal coding is obtained by fusing the target prior knowledge adapted to the target modality selected from the prior dictionary and the visual-text cross-modal consensus information extracted between the visual modality and the text modality using learnable residual parameters. The output of the current iteration serves as the input data for the next iteration, which involves selecting target prior knowledge from the prior dictionary that is compatible with the target modality and extracting visual-text cross-modal consensus information between the visual and text modalities.

20. The visual emotion recognition method according to claim 19, characterized in that, The process involves fusing the output results of multiple executions of the target prior knowledge adapted to the target modality selected from the prior dictionary and the visual-text cross-modal consensus information extracted between the visual and text modalities using learnable residual parameters, and then encoding these results to obtain the visual-text cross-modal encoding. This includes: For the video features in the i-th output result, set corresponding frame coefficients for each video frame, obtain video residual block features based on the frame coefficients and the frame features of the video frames, and fuse the video residual block features with the video features in the i-th output result to obtain the video feature fusion result corresponding to the i-th output result; For the text features in the i-th output result, set corresponding text marker coefficients for each text marker, obtain text residual block features based on the text marker coefficients and the text markers, and fuse the text residual block features with the text features in the i-th output result to obtain the text feature fusion result corresponding to the i-th output result; For the target prior knowledge in the i-th output result, set corresponding prior knowledge label coefficients for each prior knowledge label, obtain prior knowledge residual block features based on the prior knowledge label coefficients and the prior knowledge labels, and fuse the prior knowledge residual block features with the target prior knowledge in the i-th output result to obtain the prior knowledge fusion result corresponding to the i-th output result; By concatenating the video feature fusion result corresponding to the i-th output result, the text feature fusion result corresponding to the i-th output result, and the prior knowledge fusion result corresponding to the i-th output result, the feature fusion result corresponding to the i-th output result is obtained; If i is not N, then for the feature fusion result corresponding to the i-th output result, the target prior knowledge adapted to the target modality is selected from the prior dictionary and the visual-text cross-modal consensus information between the visual modality and the text modality is extracted to obtain the (i+1)-th output result; If i is N, then the feature fusion result corresponding to the i-th output result is used as the visual text cross-modal encoding.

21. A visual emotion recognition device, characterized in that, include: The first acquisition unit is used to acquire the initial visual model and sample dataset; The first training unit is used to configure a first attention head based on text sentiment evaluation coding, a second attention head based on visual text cross-modal consensus information, and a third attention head that fuses the text sentiment evaluation coding and the visual text cross-modal coding for the initial visual model. After concatenating the loss values ​​of each attention head into the model loss value of the initial visual model, the unit uses the sample dataset, the task objective of the visual sentiment recognition task to be processed, and the loss values ​​of each attention head to train the weights of each attention head in the initial visual model and output the visual sentiment recognition model. The first computing unit is used to respond to the visual emotion recognition task to be processed by calling the visual emotion recognition model to perform emotion recognition processing on the input data to be recognized, and to obtain the visual emotion recognition result. Wherein, according to the task objective of the visual emotion recognition task to be processed, the sample dataset includes image data or video data with annotated text; the visual text cross-modal consensus information is constructed through mutual information between the visual modality and the text modality; The visual text cross-modal encoding based on the visual text cross-modal consensus information is obtained through the following steps: Initialize the prior dictionary; Determine the target modality based on the task objective of the visual emotion recognition task to be processed; Based on the sample dataset, target prior knowledge that is compatible with the target modality is selected from the prior dictionary; Based on the prior knowledge of the target, cross-modal information fusion is performed on the visual modality and the text modality in the sample dataset to extract the visual-text cross-modal consensus information between the visual modality and the text modality; Cross-modal fusion encoding is performed based on the visual features, text features, and target prior knowledge corresponding to the cross-modal consensus information of the visual text to obtain the cross-modal encoding of the visual text; The target modality is one of a visual modality, a text modality, and a combined visual-text modality, and the visual feature is an image feature or a video feature; The process of fusing cross-modal information between the visual modalities and text modalities in the sample dataset based on the target prior knowledge, and extracting the visual-text cross-modal consensus information between the visual and text modalities, includes: Intramodal information integration and cross-modal information integration are performed on the visual modal and text modal in the sample dataset, respectively, to obtain the intramodal information integration result and the global consensus information integration result; Based on the intramodal information integration results and the global consensus information integration results, a text cross-attention module is constructed to search for visual information related to the text modality in the visual modality based on the target prior knowledge; Based on the intramodal information integration results and the global consensus information integration results, a visual cross-attention module is constructed to search for text information related to the visual modality in the text modality based on the target prior knowledge; Based on visual information associated with the text modality in the visual modality, textual information associated with the visual modality in the text modality, and the prior knowledge of the target, an attention module is constructed to fuse and extract the cross-modal consensus information of the visual and textual data.

22. A visual emotion recognition device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program, which, when executed by the processor, implements the steps of the visual emotion recognition method as described in any one of claims 1 to 20.

23. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the visual emotion recognition method as described in any one of claims 1 to 20.

Images