Multi-modal personality perception method and device for multiple correlation features and graph relation attention

Abstract

The invention relates to the field of computer vision, in particular to a multi-modal personality perception method and device for multiple correlation features and graph relation attention. The method comprises the following steps: acquiring an input video to be subjected to personality perception; inputting an input video into the data preprocessing module to obtain visual modal input, audio modal input and text modal input; inputting the visual modal input, the audio modal input and the text modal input into a modal feature extraction network module to obtain a scene-audio association feature, a scene-descriptor association feature, an audio-descriptor association feature and a text modal feature; inputting the scene-audio association features, the scene-descriptor association features, the audio-descriptor association features and the text modal features into a feature fusion module to obtain multi-modal fusion features; and inputting the multi-modal fusion features into a perception prediction module to obtain a personality perception result. The invention provides a multi-modal attention fusion framework which is used for personality perception.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-modal personality perception method and device with attention to multiple associated features and graph relationships. Background Art

[0002] Typically, personality includes aspects such as behavior, temperament, emotion, and psychology that define an individual's unique character. Explicit personality perception refers to the perception of a person based on the observed behavior and appearance of others, which is significantly different from actual personality (representing a person's true psychological characteristics). Personality perception aims to identify implicit personality traits in social media posts, thereby gaining insight into human behavior, emotional processes, and mental health. Personality perception helps people better understand themselves and improve their mental health, and plays an important role in areas such as psychotherapy, clinical diagnosis, career development, and social science research. In personality research, researchers have developed psychological scales and models to understand and measure personality traits. For example, the Big Five personality model measures personality through five traits: openness (O), conscientiousness (C), extraversion (E), agreeableness (A), and neuroticism (N). As an important part of social interaction, explicit personality perception significantly affects people's reactions to others, so it has important application value in areas such as interpersonal interaction analysis, human-computer interaction systems, social robots, education, marketing, and recommendation systems.

[0003] With the rise of multimedia social platforms such as YouTube and TikTok, people are increasingly using videos to share their lives. In response to this trend, ECCV (European Conference on Computer Vision) and CVPR (Conference on Computer Vision and Pattern Recognition) held competitions on personality trait perception in 2016 and 2017, generating the second version of the impression dataset. As a result, it has become more feasible to automatically perceive personality from videos. In recent years, pre-training has achieved great success in multimodal processing. For example, VL (Vision-Language) pre-trained models have shown superior performance and flexible scalability in understanding tasks such as text-visual retrieval and generation tasks such as video subtitle generation. Audio pre-trained models are able to represent complex audio information. Learning general correlations between vision, text, and audio from scratch is very computationally intensive, such as VATT (Video-Audio-Text Transformer). Therefore, an intuitive idea is to combine the most advanced visual language model with a pre-trained audio backbone. With the outstanding performance of multimodal large-scale pre-trained models in various visual downstream tasks and the emergence of graph structure learning methods, introducing them into personality perception tasks will inevitably bring unexpected results.

[0004] According to some studies, language and unconscious behaviors, including audio (voice intonation, acoustic intensity), visual cues (facial expressions, eye contact, body gestures), text (transcription of audio), and scene information are important components of the personality perception system. These multimodal features contribute significantly to the actual assessment of personality and emotion. By leveraging the complementarity of different modal information, multimodal personality perception methods can significantly enhance the robustness of perception. Therefore, various methods based on different modalities have been developed to predict personality traits. Although these research methods have achieved achievements, they still face two key challenges:

[0005] 1. Most existing methods only focus on extracting more single-modal features, ignoring the importance of correlation features between different modalities.

[0006] 2. Although there are many multimodal feature fusion methods, their actual effects still have room for improvement. Summary of the invention

[0007] In order to solve the technical problem that the existing technology ignores the correlation features between different modalities and the characteristics of the graph relationship attention network in extracting key temporal information, the embodiment of the present invention provides a multi-modal personality perception method and device with multiple correlation features and graph relationship attention. The technical solution is as follows:

[0008] On the one hand, a multimodal personality perception method based on multiple associated features and graph relationship attention is provided, the method is implemented by a multimodal personality perception device, and the method includes:

[0009] S1. Obtain an input video for personality perception.

[0010] S2. Input the input video to the data preprocessing module to obtain visual modality input, audio modality input and text modality input.

[0011] S3. Input the visual modality input, the audio modality input and the text modality input into the modality feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features and text modality features.

[0012] S4. Input the scene-audio association features, scene-description word association features, audio-description word association features and text modality features into the feature fusion module to obtain multimodal fusion features.

[0013] S5. Input the multimodal fusion features into the perception prediction module to obtain the personality perception results.

[0014] Optionally, in S3, the visual modality input, the audio modality input, and the text modality input are input into a modality feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features, and text modality features, including:

[0015] S31. Input the visual modality input and the audio modality input into a scene-audio association feature extraction module to obtain scene-audio association features.

[0016] S32: Input the visual modality input into a scene-description word association feature extraction module to obtain scene-description word association features.

[0017] S33: Input the audio modal input into an audio-description word association feature extraction module to obtain audio-description word association features.

[0018] S34. Input the text modal input into the Roberta model to obtain text modal features.

[0019] Optionally, in S31, the visual modality input and the audio modality input are input into a scene-audio correlation feature extraction module to obtain scene-audio correlation features, including:

[0020] S311. For visual modality input, use a pre-trained image encoder to extract scene image features, and construct a visual relationship graph based on the scene image features.

[0021] S312, preprocessing the audio modal input to obtain a plurality of mel-spectrogram images, extracting a plurality of audio features from the plurality of mel-spectrogram images using a pre-trained audio encoder, and constructing an audio relationship graph according to the plurality of audio features.

[0022] S313. Design a graph relationship attention network, and obtain visual relationship graph features based on the visual relationship graph and the graph relationship attention network; obtain audio relationship graph features based on the audio relationship graph and the graph relationship attention network.

[0023] S314: Concatenate the visual relationship graph features and the audio relationship graph features to obtain scene-audio association features.

[0024] Optionally, the step of inputting the visual modality input into a scene-description word association feature extraction module in S32 to obtain scene-description word association features includes:

[0025] S321. Construct a list of personality descriptive words.

[0026] S322: Generate personality description word embedding according to the personality description word list and the text encoder.

[0027] S323: Generate scene image embedding based on the visual modality input and the image encoder.

[0028] S324. Calculate the dot product of the personality description word embedding and the scene image embedding, and then obtain the scene-description word association feature.

[0029] Optionally, the step of inputting the audio modal input into an audio-description word association feature extraction module in S33 to obtain an audio-description word association feature includes:

[0030] S331. Generate personality description word embedding according to the personality description word list and the text encoder.

[0031] S332: Generate audio embedding according to the audio modality input and the pre-trained audio encoder.

[0032] S333. Calculate the dot product of the personality description word embedding and the audio embedding, and then obtain the audio-description word association feature.

[0033] Optionally, in S4, the scene-audio association feature, the scene-description word association feature, the audio-description word association feature, and the text modality feature are input into a feature fusion module to obtain a multimodal fusion feature, including:

[0034] S41. Calculate the scene-audio association features, scene-description word association features, audio-description word association features and text modal features through the first attention score calculation module, the second attention score calculation module, the third attention score calculation module and the fourth attention score calculation module, respectively, to obtain the scene-audio association feature self-attention score, the scene-description word association feature self-attention score, the audio-description word association feature self-attention score and the text modal feature self-attention score.

[0035] S42. According to the scene-audio associated feature self-attention score, the scene-description word associated feature self-attention score, the audio-description word associated feature self-attention score and the text modality feature self-attention score, calculate the normalized value of the scene-audio associated feature self-attention score, the normalized value of the scene-description word associated feature self-attention score, the normalized value of the audio-description word associated feature self-attention score and the normalized value of the text modality feature self-attention score.

[0036] S43. Calculate the multimodal fusion feature according to the normalized value of the self-attention score of the scene-audio associated feature, the normalized value of the self-attention score of the scene-description word associated feature, the normalized value of the self-attention score of the audio-description word associated feature, and the normalized value of the self-attention score of the text modality feature.

[0037] On the other hand, a multimodal personality perception device for multiple associated features and graph relationship attention is provided, and the device is applied to a multimodal personality perception method for multiple associated features and graph relationship attention, and the device comprises:

[0038] The acquisition module is used to acquire the input video to be subjected to personality perception.

[0039] The data preprocessing module is used to input the input video into the data preprocessing module to obtain visual modality input, audio modality input and text modality input.

[0040] The modal feature extraction network module is used to input the visual modal input, the audio modal input and the text modal input into the modal feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features and text modal features.

[0041] The feature fusion module is used to input scene-audio association features, scene-description word association features, audio-description word association features and text modality features into the feature fusion module to obtain multimodal fusion features.

[0042] The output module is used to input the multimodal fusion features into the perception prediction module to obtain the personality perception results.

[0043] Optionally, the modal feature extraction network module is further used to:

[0044] S31. Input the visual modality input and the audio modality input into a scene-audio association feature extraction module to obtain scene-audio association features.

[0045] S32: Input the visual modality input into a scene-description word association feature extraction module to obtain scene-description word association features.

[0046] S33: Input the audio modal input into an audio-description word association feature extraction module to obtain audio-description word association features.

[0047] S34. Input the text modal input into the Roberta model to obtain text modal features.

[0048] Optionally, the modal feature extraction network module is further used to:

[0049] S311. For visual modality input, use a pre-trained image encoder to extract scene image features, and construct a visual relationship graph based on the scene image features.

[0050] S312, preprocessing the audio modal input to obtain a plurality of mel-spectrogram images, extracting a plurality of audio features from the plurality of mel-spectrogram images using a pre-trained audio encoder, and constructing an audio relationship graph according to the plurality of audio features.

[0051] S313. Design a graph relationship attention network, and obtain visual relationship graph features based on the visual relationship graph and the graph relationship attention network; obtain audio relationship graph features based on the audio relationship graph and the graph relationship attention network.

[0052] S314: Concatenate the visual relationship graph features and the audio relationship graph features to obtain scene-audio association features.

[0053] Optionally, the modal feature extraction network module is further used to:

[0054] S321. Construct a list of personality descriptive words.

[0055] S322: Generate personality description word embedding according to the personality description word list and the text encoder.

[0056] S323: Generate scene image embedding based on the visual modality input and the image encoder.

[0057] S324. Calculate the dot product of the personality description word embedding and the scene image embedding, and then obtain the scene-description word association feature.

[0058] Optionally, the modal feature extraction network module is further used to:

[0059] S331. Generate personality description word embedding according to the personality description word list and the text encoder.

[0060] S332: Generate audio embedding according to the audio modality input and the pre-trained audio encoder.

[0061] S333. Calculate the dot product of the personality description word embedding and the audio embedding, and then obtain the audio-description word association feature.

[0062] Optionally, the feature fusion module is further used to:

[0063] S41. Calculate the scene-audio association features, scene-description word association features, audio-description word association features and text modal features through the first attention score calculation module, the second attention score calculation module, the third attention score calculation module and the fourth attention score calculation module, respectively, to obtain the scene-audio association feature self-attention score, the scene-description word association feature self-attention score, the audio-description word association feature self-attention score and the text modal feature self-attention score.

[0064] S42. According to the scene-audio associated feature self-attention score, the scene-description word associated feature self-attention score, the audio-description word associated feature self-attention score and the text modality feature self-attention score, calculate the normalized value of the scene-audio associated feature self-attention score, the normalized value of the scene-description word associated feature self-attention score, the normalized value of the audio-description word associated feature self-attention score and the normalized value of the text modality feature self-attention score.

[0065] S43. Calculate the multimodal fusion feature according to the normalized value of the self-attention score of the scene-audio associated feature, the normalized value of the self-attention score of the scene-description word associated feature, the normalized value of the self-attention score of the audio-description word associated feature, and the normalized value of the self-attention score of the text modality feature.

[0066] On the other hand, a multimodal personality perception device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the multimodal personality perception methods of multiple association features and graph relationship attention is implemented.

[0067] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned multimodal personality perception methods of multiple association features and graph relationship attention.

[0068] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0069] In an embodiment of the present invention, a multimodal attention fusion framework is proposed for personality perception. The framework utilizes a variety of correlation features and a graph relation attention network, and is designed for visual-language-audio processing. The present invention includes audio modalities in a unified manner by extending the visual-language pre-training model CLIP. The present invention utilizes contrastive learning techniques from the perspectives of cross-modality and internal modality to make full use of the multimodal pre-training model. In order to extract scene-audio correlation features, a scene-audio correlation feature extraction method is proposed in the framework of the present invention. In addition, the present invention extends the CLIP model to CLIPAudio audio, which is used to mine the correlation features of scene personality descriptors and audio personality descriptions, respectively. The present invention also utilizes the pre-trained Roberta model to extract text-specific features. In order to effectively fuse these multimodal information, the present invention designs a multimodal attention feature fusion strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0071] Figure 1 It is a flow chart of a multi-modal personality perception method with multiple associated features and graph relationship attention provided by an embodiment of the present invention;

[0072] Figure 2 It is an overall architecture diagram provided by an embodiment of the present invention;

[0073] Figure 3 is a flow chart of a scene audio correlation feature extraction method provided by an embodiment of the present invention;

[0074] Figure 4 It is a graph relationship attention network graph provided by an embodiment of the present invention;

[0075] Figure 5 is a flow chart of a method for extracting associated features of scene and audio personality descriptions provided by an embodiment of the present invention;

[0076] Figure 6is a flow chart of a text embedding feature extraction method based on Roberta provided in an embodiment of the present invention;

[0077] Figure 7 It is a flow chart of a multi-modal attention feature fusion strategy provided by an embodiment of the present invention;

[0078] Figure 8 It is a block diagram of a multi-modal personality perception device with multiple associated features and graph relationship attention provided by an embodiment of the present invention;

[0079] Fig. 9 It is a structural schematic diagram of a multimodal personality perception device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0081] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0082] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0083] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0084] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0085] The embodiment of the present invention provides a multi-modal personality perception method based on multiple associated features and graph relationship attention. The method can be implemented by a multi-modal personality perception device, which can be a terminal or a server. Figure 1 The flowchart of the multimodal personality perception method of multiple associated features and graph relationship attention is shown in the figure. The processing flow of the method may include the following steps:

[0086] S1. Obtain an input video for personality perception.

[0087] S2. Input the input video to the data preprocessing module to obtain visual modality input, audio modality input and text modality input.

[0088] S3. Input the visual modality input, the audio modality input and the text modality input into the modality feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features and text modality features.

[0089] S4. Input the scene-audio association features, scene-description word association features, audio-description word association features and text modality features into the feature fusion module to obtain multimodal fusion features.

[0090] S5. Input the multimodal fusion features into the perception prediction module to obtain the personality perception results.

[0091] In view of the problems of the prior art, the present invention proposes a multimodal personality perception framework (CLIP for Vision, Language, and Audio), which expands CLIP (Contrastive Language-lmage Pre-Training) to adapt to the audio modality and adopts a unified triple encoder structure for multimodal processing. The framework proposed by the present invention integrates various methods for learning high-level feature representations of multimodality. Specifically, the present invention first adopts the leading visual-language model CLIP as a visual and text encoder. In order to ensure the consistency and efficiency of training, the present invention uses an audio encoder with a similar structure to the visual encoder. The present invention proposes a method to extract scene-audio association features. In order to simultaneously encode visual scenes, audio information and personality descriptors, the present invention designs the CLIPAudio method to extract scene-personality descriptor association and audio-personality descriptor association features. In order to better utilize the multimodal representation learned by the framework of the present invention, the present invention further uses the pre-trained Roberta model to extract text features. In addition, in order to better focus on important features and reduce redundant features, the present invention designs a multimodal attention feature fusion strategy for fusing multiple features and training an MLP (Multilayer Perceptron) model to predict personality traits. For effective modeling, a two-stage training process is adopted. First, the present invention trains each modality-specific network separately. Subsequently, the present invention combines these sub-networks and fine-tunes the entire framework in a multimodal manner.

[0092] Furthermore, recently, GNNs (Graph Neural Networks) have been widely used in various challenging tasks due to their ability to model data generated from non-Euclidean domains and capture the internal dependencies of the data. The main advantage of GCNs (Graph Convolution Networks) is that they can learn latent embeddings of nodes directly based on neighbor nodes and their relationships, and have achieved widespread success. They have been intensively studied for exploring relational data structures, contextual information, and geometric dependencies. Similar to the convolution operation in CNNs (Convolutional Neural Networks) that aggregates information from neighboring pixels from a predefined grid cell, graph convolution operations aggregate information from neighboring vertices within a predefined graph distance. In addition to applying GCNs on landmark-based facial data for FER (Facial Expression Recognition), there are also a large number of studies that apply GCNs to explore the correlation between different frames in a video for video-based FER, and the correlation between different facial AUs (Action Units) for FER. However, the above works have not considered the attention mechanism in GCNs and CNN-based learning, which can model scene images or log-mel spectrograms as images for personality prediction. Therefore, this paper proposes a graph relation attention network based on CLIP's ViT encoder and VGGish CNN for personality prediction.

[0093] Conceptually, the CLIP model consists of two subnetworks: a text encoder and an image encoder. These two parts of the CLIP model are jointly pre-trained with natural language supervision. This training setup enables the model to generalize its classification capabilities to image samples belonging to previously unseen datasets, based on the provided labels without additional fine-tuning. For the text encoding part, a slightly modified Transformer architecture was adopted. For the chosen 12-layer model, the input text is encoded using lowercase byte pairs with a vocabulary of size 49408. Due to computational constraints, the maximum sequence length is limited to 76. Two different architectures are considered for the image encoding part of the CLIP model. One of them is ViT (Vision Transformer), whose architecture makes it similar to a text header. Multimodal designs were first applied in video-related tasks and later adapted to sound classification tasks. However, despite the multimodal design, these methods use at most two modalities simultaneously, while recent studies have shown that using more modalities is beneficial. The above multimodal methods share a key idea of ​​contrastive learning. This technique belongs to the branch of self-supervised learning and helps overcome the problem of lack of high-quality labeled data. Therefore, the present invention proposes an extension of the CLIP model that can process audio data in addition to text and images.

[0094] like Figure 2 As shown in the figure, the framework proposed in this paper aims to predict the personality traits of individuals based on short videos. The framework includes four main steps: (1) data preprocessing; (2) modal feature extraction network; (3) feature fusion and model training; (4) perception prediction. Specifically:

[0095] S1. Obtain an input video for personality perception.

[0096] S2. Input the input video to the data preprocessing module to obtain visual modality input, audio modality input and text modality input.

[0097] In one feasible implementation, during the data preprocessing stage, the input video is segmented into multiple modal inputs, including visual, audio, and text. The input short video covers three natural modalities: the visual modality is used for image sequences, the audio modality is used for speech, and the text modality is used for transcripts and personality descriptors. During the feature extraction stage, the framework of the present invention uses modality-specific networks to learn personality trait information from each input modality separately. Different modality-specific features are obtained through the respective specifically learned networks, which are fused through the multimodal attention fusion strategy for different modality features, and then input into the regression head to predict the final scores of various personality traits.

[0098] S3. Input the visual modality input, the audio modality input and the text modality input into the modality feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features and text modality features.

[0099] In a feasible implementation manner, the above step S3 mainly extracts features from three types of multimodal association features by the method proposed by the present invention, which are: using a method for extracting scene-audio association features, using a CLIPAudio method to extract scene-description word association features and audio-description word association features. Next, the present invention will elaborate on these feature extraction methods as follows:

[0100] S31. Input the visual modality input and the audio modality input into a scene-audio association feature extraction module to obtain scene-audio association features.

[0101] Figure 3 The framework of the scene-audio correlation feature extraction method of the present invention is described, which consists of three parts: 1) construction of visual and audio relationship graph; 2) graph relationship attention network; 3) extraction of scene-audio correlation features.

[0102] Specifically, the above step S31 may include the following steps S311-S314:

[0103] S311. For visual modality input, use a pre-trained image encoder to extract scene image features, and construct a visual relationship graph based on the scene image features.

[0104] S312, preprocessing the audio modal input to obtain a plurality of mel-spectrogram images, extracting a plurality of audio features from the plurality of mel-spectrogram images using a pre-trained audio encoder, and constructing an audio relationship graph according to the plurality of audio features.

[0105] In a feasible implementation, in order to better describe the acquisition of personality-related clues from extracted scene and audio features, the present invention proposes a new method, the structure of which is as follows: Figure 3 As shown, it is used to extract scene-audio association features.

[0106] Specifically, a pre-trained CLIP image encoder ViT is used to extract scene image features SI from the scene image. For audio data, the present invention pre-processes the audio data into multiple Mel-spectrogram images, extracts audio features through a pre-trained VGGish CNN model, and obtains multiple audio features MA. Then, the present invention uses these scene image features SI to construct a visual relationship graph VRG.

[0107] In the present invention, each scene image feature is regarded as a node in a complete graph, and the so-called complete graph means that all nodes are connected to each other. Similarly, the present invention can use multiple audio features MA to construct an audio relationship graph ARG. For these two relationship graphs, the present invention uses the cosine similarity between two node features to describe the edge attributes of each relationship graph. Assume that for a relationship graph, the features of any two nodes are , whose adjacent nodes are , F is the length of the node feature, then the edge attributes of the two connected nodes are as follows:

[0108] (1)

[0109] S313. Design a graph relationship attention network, and obtain visual relationship graph features based on the visual relationship graph and the graph relationship attention network; obtain audio relationship graph features based on the audio relationship graph and the graph relationship attention network.

[0110] In a feasible implementation, in order to explore the association between scene image features and multiple audio features, the present invention designs a graph relation attention network for feature transfer and aggregation of VRG and ARG, such as Figure 4 As shown. It consists of two graph relation attention convolution modules, global maximum pooling, global average pooling, connection operation, one-dimensional convolution module and residual connection. Each graph relation attention convolution module includes a graph attention convolution layer, a RELU activation layer and a one-dimensional convolution module. Among them, the residual connection is used to stabilize the learning process. The global average pooling layer and the global maximum pooling layer are used to independently compress the output features of the second one-dimensional convolution layer into features without time dimension. In the present invention, GAT (GraphAttention Network) is used as the basis of the graph attention convolution layer, which is defined as Next, the present invention extracts two relationship graph features based on this, as follows:

[0111] (2)

[0112] (3)

[0113] (4)

[0114] In the formula, is the input node feature, is the relu activation function, is a one-dimensional CNN convolution function, and are the weight matrix and bias term of CNN. and Represent global maximum pooling and global average pooling respectively, Represents feature concatenation. Finally, the final feature output of the graph relation attention network is:

[0115] (5)

[0116] S314: Concatenate the visual relationship graph features and the audio relationship graph features to obtain scene-audio association features.

[0117] In a feasible implementation, the present invention concatenates the above two features in the second dimension and fuses these features through a one-dimensional convolutional layer. Finally, the present invention concatenates the output features of the visual and audio relationship streams and further integrates the information through a multi-layer perceptron (MLP), including a linear layer, a Relu activation layer, and a dropout layer. Ultimately, the present invention can obtain the correlation features between the scene image and the audio to represent information related to personality.

[0118] S32: Input the visual modality input into a scene-description word association feature extraction module to obtain scene-description word association features.

[0119] In a feasible implementation manner, the structure of the CLIPAudio method of the present invention is as follows: Figure 5 As shown in Figure 1, it includes scene-description word association feature extraction and audio-description word association feature extraction. The CLIPAudio method consists of three parts: 1) personality descriptor list construction; 2) scene-personality descriptor association feature extraction; 3) audio-personality descriptor association feature extraction.

[0120] Here, we use CLIP's text encoder to generate text description embeddings for these personality descriptors. At the same time, we use the "openai / clip-vit-large-patch14-336" model as CLIP's image encoder to generate scene embeddings from videos; and use the pre-trained VGGish CNN as the audio encoder to generate audio embeddings from the audio modality of the video.

[0121] Optionally, the above step S32 may include the following steps S321-S324:

[0122] S321. Construct a personality description list.

[0123] In one feasible implementation, the present invention generates personality descriptor embeddings using a personality descriptor list, which includes 1,608 adjectives that can accurately describe human personality traits. This list contains personality terms carefully selected from personality surveys [37,38] and multiple dictionaries [39,40,41] to ensure the diversity and representativeness of the descriptions, thereby effectively capturing the complexity of human personality.

[0124] S322: Generate personality description word embedding according to the personality description word list and the text encoder.

[0125] S323: Generate scene image embedding based on the visual modality input and the image encoder.

[0126] S324. Calculate the dot product of the personality description word embedding and the scene image embedding, and then obtain the scene-description word association feature.

[0127] In a feasible implementation, Figure 5 As shown in the upper part of the figure, the present invention proposes CLIP-PersD (a part of the present CLIPAudio method) to extract scene features and personality descriptor features. It includes three main steps: (1) using CLIP's image encoder to generate embeddings for scene images; (2) using CLIP's text encoder to generate embeddings for personality descriptors; (3) calculating the similarity score between the scene embedding and the text embedding by calculating their dot product.

[0128] Specifically, first, suppose Indicates that it contains A collection of personality descriptors. Extracted from the video scene image collection. For each personality descriptor, there is , this paper uses CLIP's text encoder to generate text embedding ,in At the same time, for each image , the present invention uses CLIP's image encoder to generate image embedding Each image corresponds to a The feature vector of the personality descriptor, where each value represents the similarity score between the image embedding and the corresponding personality descriptor embedding. Here, the present invention calculates their similarity scores by calculating their dot product. Specifically, the present invention obtains a feature matrix , where each entry Indicates Personality descriptors and The similarity score between the images is calculated as the dot product of their corresponding personality descriptors and the image embedding:

[0129] (6)

[0130] Finally, in order to obtain the The present invention needs to average the feature matrices of all images in the video to obtain a single feature vector of the video:

[0131] (7)

[0132] in, It is The feature vector of the video, It is The number of images extracted from a video. In the present invention, .

[0133] S33: Input the audio modal input into an audio-description word association feature extraction module to obtain audio-description word association features.

[0134] In a feasible implementation, Figure 5 As shown in the lower part of the figure, the present invention proposes a new method named CLIPVGGish for extracting audio description word related features based on CLIP and pre-trained VGGish CNN. The method includes three main steps: (1) using the text encoder of CLIP to generate the embedding of personality description words; (2) using the VGGish CNN encoder to generate the audio embedding of audio clues; (3) calculating the similarity score between the personality description word embedding and the audio embedding by calculating the dot product between them.

[0135] Optionally, the above step S33 may include the following steps S331-S333:

[0136] S331. Generate personality description word embedding according to the personality description word list and the text encoder.

[0137] S332: Generate audio embedding according to the audio modality input and the pre-trained audio encoder.

[0138] S333. Calculate the dot product of the personality description word embedding and the audio embedding, and then obtain the audio-description word association feature.

[0139] In a feasible implementation, the present invention can generate embeddings TP for all personality descriptors P. In addition, the present invention first converts the video into audio data using the ffm-peg tool. After a series of operations such as frame processing, pre-emphasis, window function, fast Fourier transform, Mel filtering, and logarithmic transformation, the audio data is converted into a logarithmic Mel spectrogram. In the present invention, for approximately 15 seconds of video, 25 evenly spaced logarithmic Mel spectrograms M with a shape of [60×64] are extracted. In addition, a VGGish CNN pre-trained on YouTube's AudioSet data is used and fine-tuned as an audio encoder. These Mel spectrograms are input into the pre-trained VGGish CNN model to generate an audio embedding AM.

[0140] Furthermore, each audio embedding corresponds to a feature vector containing the values ​​of N personality descriptors, where each value corresponds to an audio embedding The present invention calculates the similarity score between the audio embedding TP and the corresponding personality description word embedding TP. and the dot product of the description word embedding TP to calculate their similarity scores. Specifically, the present invention can obtain a feature matrix , where each entry Indicates Personality Descriptors and The similarity score between the audio embeddings. It is calculated as the corresponding personality descriptor The dot product with the audio embedding AM:

[0141] (8)

[0142] Finally, in order to obtain the The present invention averages the feature matrices of all audio embeddings in a video:

[0143] (9)

[0144] Here, It is The feature vector of the video, where |AM| is the The number of audio embeddings extracted from a video. In the present invention, |AM|=25 is set.

[0145] Furthermore, in order to reduce the dimensions of the above scene personality descriptor associated features and audio personality descriptor associated features, the present invention constructs and trains two MLP regression models, each of which contains two fully connected (FC) layers. Here, each FC layer includes a linear layer, a relu activation layer, and a dropout layer (parameter p=0.3), which is used to reduce the dimension of the feature from 1608 to 512, and finally to 128. Each linear layer associated with the scene personality descriptor associated features has an additional Bn layer.

[0146] S34. Input the text modal input into the Roberta model to obtain text modal features.

[0147] In one feasible implementation, the present invention uses a pre-trained Roberta model to extract embedded feature vectors from audio transcripts. The model uses the parameters of the Hugging Face pre-trained model and has been fine-tuned on a personality prediction dataset. Next, the present invention will introduce the data preprocessing and model architecture of the Roberta model.

[0148] 1) Roberta preprocessing: Roberta is a variant of the BERT model, using the Transformer architecture, with 12 encoder layers and a 12-head self-attention mechanism. Through unsupervised learning, Roberta learns language representation on a larger scale than BERT. It is worth noting that Roberta is better able to grasp context and long-distance dependencies.

[0149] 2) Model architecture: In the method of the present invention, text processing relies on RobertaTokenizer, which is based on the pre-trained Roberta model. Tokenizer performs several key steps, including: (1) Preprocessing: converting text to lowercase, removing extra spaces, etc.; (2) Word segmentation: Tokenizer segments text into tokens based on the constructed word list; (3) Adding special tags: In order to mark the beginning and end of the text, special tags such as [CLS] and [SEP] are added. After these steps, the token sequence is converted into a token index representation. The process of extracting text embedding features using Roberta is as follows: Figure 6 shown.

[0150] S4. Input the scene-audio association features, scene-description word association features, audio-description word association features and text modality features into the feature fusion module to obtain multimodal fusion features.

[0151] like Figure 7 A multimodal attention feature fusion strategy and a regression head based on an MLP model are demonstrated.

[0152] First, before all features are input to the multimodal attention feature fusion module, an FC layer is used to ensure dimensionality consistency. For scene description related features, the FC layer structure is slightly different, including a linear layer, a BN layer, a relu activation layer, and a dropout layer with p=0.3. For other input features, the FC layer structure is the same, including a linear layer, a relu activation layer, and a dropout layer with p=0.2.

[0153] Optionally, the above step S4 may include the following steps S41-S43:

[0154] S41. Calculate the scene-audio association features, scene-description word association features, audio-description word association features and text modal features through the first attention score calculation module, the second attention score calculation module, the third attention score calculation module and the fourth attention score calculation module, respectively, to obtain the scene-audio association feature self-attention score, the scene-description word association feature self-attention score, the audio-description word association feature self-attention score and the text modal feature self-attention score.

[0155] S42. According to the scene-audio associated feature self-attention score, the scene-description word associated feature self-attention score, the audio-description word associated feature self-attention score and the text modality feature self-attention score, calculate the normalized value of the scene-audio associated feature self-attention score, the normalized value of the scene-description word associated feature self-attention score, the normalized value of the audio-description word associated feature self-attention score and the normalized value of the text modality feature self-attention score.

[0156] S43. Calculate the multimodal fusion feature according to the normalized value of the self-attention score of the scene-audio associated feature, the normalized value of the self-attention score of the scene-description word associated feature, the normalized value of the self-attention score of the audio-description word associated feature, and the normalized value of the self-attention score of the text modality feature.

[0157] In a feasible implementation, the present invention designs a multimodal attention feature fusion strategy to fuse four features. First, four attention score calculation modules with the same structure (parameters are not shared) are used to obtain the self-attention scores of four different features. Then, the normalized values ​​of the attention scores of the four different features are obtained by dividing each self-attention score by the sum of the self-attention scores of all modalities. Finally, the attention scores are multiplied by the feature vectors of different features and summed to form the final multimodal fusion features. Through the proposed strategy, the present invention aims to capture the importance based on the complex relationship between different modal features. The feature attention module includes a fully connected layer with a linear layer and a tanh function, and another layer with a linear layer and a sigmoid function. Assume that the obtained specific modality representation vector is ,but The self-attention score is recorded as For the four features, their self-attention scores can be calculated as follows:

[0158] (10)

[0159] in, and Represent the learnable parameters of the two fully connected layers (parameters between different modalities are not shared). The tanh and sigmoid functions are used to limit the attention weights to the interval [-1, 1]. Then, the obtained attention vector (* represents any one of {T, VT, AT, VA}) and performs element-wise multiplication with each dimension of the multimodal fusion feature head h. The formula is as follows:

[0160] (11)

[0161] (12)

[0162] (13)

[0163] Finally, in order to obtain more robust multimodal fusion features, the present invention uses a multi-head mechanism to generate multiple fusion features and connects them to form the final multimodal fusion feature. :

[0164] (14)

[0165] S5. Input the multimodal fusion features into the perception prediction module to obtain the personality perception results.

[0166] In the present invention, a two-layer perceptron (MLP) regression model is used as a model for personality perception. This perception model includes an FC module and an MLP module. Here, the FC module consists of a linear layer, a relu activation layer, and a dropout layer to reduce the dimension of the multimodal fusion feature vector. Subsequently, the multimodal fusion feature vector is processed by a single-layer MLP model to predict the scores of different personality traits, and its output value is between 0 and 1. Here, the MLP model consists of a linear layer and a sigmoid activation layer.

[0167] Furthermore, in order to optimize the overall framework, the present invention uses the Bell loss function, which is defined as follows:

[0168] (15)

[0169] in, and are the true label and the predicted value respectively, is the derivative parameter that controls the magnitude of the change. resulting in a higher gradient. is a scale parameter that controls the range of the loss function. In the present invention, , .

[0170] Using the multimodal personality perception method based on multiple correlation features and graph relationship attention network disclosed in the present invention, the system can intelligently and quickly predict people's Big Five personality traits. The method can be widely used in job interviews, purchasing behavior, national health (the pathogenesis and rehabilitation of mental illnesses such as depression and anxiety), education, talent recruitment, crime prevention, etc.

[0171] Automatic personality trait perception uses multimodal information to accurately predict personality traits, such as behavior and appearance, from videos. On the one hand, existing multimodal large-scale pre-trained models perform well in downstream tasks. On the other hand, existing methods ignore the correlation features between different modalities and the characteristics of graph relational attention networks in extracting key temporal information. Therefore, the present invention proposes a new multimodal framework based on multiple correlation features and graph relational attention networks for personality assessment. The present invention proposes a method for extracting scene-audio correlation features through graph relational representation learning. At the same time, the present invention designs a CLIPAudio method that extends CLIP to extract correlation features between scene personality descriptors and audio personality descriptors. Specifically, the present invention explores the correlation between personality descriptions and visual and audio modalities using cross-modal and intramodal similarities. At the same time, the present invention uses the pre-trained RoBERTa model as the backbone of a text modality-specific network to extract text clues related to personality traits. Finally, the present invention designs a multimodal attention feature fusion strategy to fuse them and train a regression model to predict personality traits.

[0172] In an embodiment of the present invention, a multimodal attention fusion framework is proposed for personality perception. The framework utilizes multiple association features and graph relation attention networks and is designed for visual-language-audio processing. The present invention includes audio modalities in a unified manner by extending the visual-language pre-training model CLIP. The present invention utilizes contrastive learning techniques from the perspectives of cross-modality and internal modality to make full use of the multimodal pre-training model. In order to extract scene-audio association features, a scene-audio association feature extraction method is proposed in the framework of the present invention. In addition, the present invention extends the CLIP model to CLIPAudio audio, which is used to mine the association features of scene personality descriptors and audio personality descriptors, respectively. The present invention also utilizes the pre-trained Roberta model to extract text-specific features. In order to effectively fuse these multimodal information, the present invention designs a multimodal attention feature fusion strategy.

[0173] Figure 8 This is a block diagram of a multi-modal personality perception device with multiple associated features and graph relationship attention according to an exemplary embodiment. The device is used in a multi-modal personality perception method with multiple associated features and graph relationship attention. Figure 8 The device includes an acquisition module 310, a data preprocessing module 320, a modal feature extraction network module 330, a feature fusion module 340 and an output module 350. Among them:

[0174] The acquisition module 310 is used to acquire an input video to be subjected to personality perception.

[0175] The data preprocessing module 320 is used to input the input video into the data preprocessing module to obtain visual modality input, audio modality input and text modality input.

[0176] The modal feature extraction network module 330 is used to input the visual modal input, the audio modal input and the text modal input into the modal feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features and text modal features.

[0177] The feature fusion module 340 is used to input the scene-audio association features, the scene-description word association features, the audio-description word association features and the text modality features into the feature fusion module to obtain multimodal fusion features.

[0178] The output module 350 is used to input the multimodal fusion features into the perception prediction module to obtain the personality perception result.

[0179] In an embodiment of the present invention, a multimodal attention fusion framework is proposed for personality perception. The framework utilizes multiple association features and graph relation attention networks and is designed for visual-language-audio processing. The present invention includes audio modalities in a unified manner by extending the visual-language pre-training model CLIP. The present invention utilizes contrastive learning techniques from the perspectives of cross-modality and internal modality to make full use of the multimodal pre-training model. In order to extract scene-audio association features, a scene-audio association feature extraction method is proposed in the framework of the present invention. In addition, the present invention extends the CLIP model to CLIPAudio audio, which is used to mine the association features of scene personality descriptors and audio personality descriptors, respectively. The present invention also utilizes the pre-trained Roberta model to extract text-specific features. In order to effectively fuse these multimodal information, the present invention designs a multimodal attention feature fusion strategy.

[0180] Fig. 9 is a schematic diagram of the structure of a multimodal personality perception device provided by an embodiment of the present invention, such as Fig. 9 As shown, the multimodal personality perception device may include the above Figure 8 The multi-modal personality perception device 410 of the plurality of associated features and graph relationship attention shown. Optionally, the multi-modal personality perception device 410 may include a first processor 2001.

[0181] Optionally, the multimodal personality perception device 410 may further include a memory 2002 and a transceiver 2003 .

[0182] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0183] Combine the following Fig. 9The components of the multimodal personality perception device 410 are specifically introduced as follows:

[0184] The first processor 2001 is the control center of the multimodal personality perception device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0185] Optionally, the first processor 2001 may perform various functions of the multimodal personality perception device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0186] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Fig. 9 CPU0 and CPU1 are shown in FIG.

[0187] In a specific implementation, as an embodiment, the multimodal personality perception device 410 may also include multiple processors, such as Fig. 9 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0188] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0189] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Fig. 9 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0190] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0191] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Fig. 9 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0192] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Fig. 9 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0193] It should be noted that Fig. 9 The structure of the multimodal personality perception device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0194] In addition, the technical effects of the multimodal personality perception device 410 can refer to the technical effects of the multimodal personality perception method with multiple associated features and graph relationship attention described in the above method embodiment, which will not be repeated here.

[0195] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0196] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0197] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal personality perception method based on multiple correlation features and graph relationship attention, characterized in that: The method comprises: S1, obtaining an input video for personality perception; S2, inputting the input video into a data preprocessing module to obtain a visual modality input, an audio modality input, and a text modality input; S3, inputting the visual modality input, the audio modality input and the text modality input into a modality feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features and text modality features; S4, inputting the scene-audio association feature, the scene-description word association feature, the audio-description word association feature and the text modality feature into a feature fusion module to obtain a multimodal fusion feature; S5. Input the multimodal fusion features into a perception prediction module to obtain a personality perception result.

2. The multimodal personality perception method based on multiple association features and graph relationship attention according to claim 1 is characterized in that: The step of inputting the visual modality input, the audio modality input, and the text modality input into a modality feature extraction network module in S3 to obtain scene-audio association features, scene-description word association features, audio-description word association features, and text modality features includes: S31, inputting the visual modality input and the audio modality input into a scene-audio association feature extraction module to obtain a scene-audio association feature; S32, inputting the visual modality input into a scene-description word association feature extraction module to obtain a scene-description word association feature; S33, inputting the audio modal input into an audio-description word association feature extraction module to obtain an audio-description word association feature; S34, inputting the text modality input into the Roberta model to obtain text modality features.

3. The multimodal personality perception method based on multiple association features and graph relationship attention according to claim 2 is characterized in that: The step of inputting the visual modality input and the audio modality input into a scene-audio correlation feature extraction module in S31 to obtain scene-audio correlation features includes: S311, extracting scene image features using a pre-trained image encoder for the visual modality input, and constructing a visual relationship graph according to the scene image features; S312, preprocessing the audio modality input to obtain a plurality of mel-spectrogram images, extracting a plurality of audio features from the plurality of mel-spectrogram images using a pre-trained audio encoder, and constructing an audio relationship graph according to the plurality of audio features; S313, designing a graph relationship attention network, obtaining visual relationship graph features according to the visual relationship graph and the graph relationship attention network; obtaining audio relationship graph features according to the audio relationship graph and the graph relationship attention network; S314: Concatenate the visual relationship graph features and the audio relationship graph features to obtain scene-audio association features.

4. The multimodal personality perception method based on multiple association features and graph relationship attention according to claim 2 is characterized in that: The step of inputting the visual modality input into a scene-description word association feature extraction module in S32 to obtain scene-description word association features includes: S321, construct a list of personality descriptors; S322, generating a personality description word embedding according to the personality description word list and the text encoder; S323, generating a scene image embedding according to the visual modality input and the image encoder; S324: Calculate the dot product of the personality description word embedding and the scene image embedding, and then obtain the scene-description word association feature.

5. The multimodal personality perception method based on multiple association features and graph relationship attention according to claim 2 is characterized in that: The step of inputting the audio modal input into an audio-description word association feature extraction module in S33 to obtain an audio-description word association feature includes: S331, generating personality description word embedding according to the personality description word list and the text encoder; S332, generating an audio embedding according to the audio modality input and a pre-trained audio encoder; S333. Calculate the dot product of the personality description word embedding and the audio embedding, and then obtain the audio-description word association feature.

6. The multimodal personality perception method based on multiple association features and graph relationship attention according to claim 1 is characterized in that: The step S4 of inputting the scene-audio association feature, the scene-description word association feature, the audio-description word association feature and the text modality feature into a feature fusion module to obtain a multimodal fusion feature includes: S41, respectively calculating the scene-audio association feature, the scene-description word association feature, the audio-description word association feature and the text modality feature through the first attention score calculation module, the second attention score calculation module, the third attention score calculation module and the fourth attention score calculation module to obtain the scene-audio association feature self-attention score, the scene-description word association feature self-attention score, the audio-description word association feature self-attention score and the text modality feature self-attention score; S42, according to the scene-audio associated feature self-attention score, the scene-description word associated feature self-attention score, the audio-description word associated feature self-attention score and the text modality feature self-attention score, calculate a normalized value of the scene-audio associated feature self-attention score, a normalized value of the scene-description word associated feature self-attention score, a normalized value of the audio-description word associated feature self-attention score and a normalized value of the text modality feature self-attention score; S43. Calculate the multimodal fusion feature according to the normalized value of the scene-audio associated feature self-attention score, the normalized value of the scene-description word associated feature self-attention score, the normalized value of the audio-description word associated feature self-attention score, and the normalized value of the text modality feature self-attention score.

7. A multi-modal personality perception device with multiple associated features and graph relationship attention, the multi-modal personality perception device with multiple associated features and graph relationship attention is used to implement the multi-modal personality perception method with multiple associated features and graph relationship attention as claimed in any one of claims 1 to 6, characterized in that: The device comprises: An acquisition module, used for acquiring an input video to be subjected to personality perception; A data preprocessing module, used for inputting the input video into the data preprocessing module to obtain visual modality input, audio modality input and text modality input; A modal feature extraction network module, used for inputting the visual modal input, the audio modal input and the text modal input into the modal feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features and text modal features; A feature fusion module, used for inputting the scene-audio association feature, the scene-description word association feature, the audio-description word association feature and the text modality feature into the feature fusion module to obtain a multimodal fusion feature; The output module is used to input the multimodal fusion features into the perception prediction module to obtain the personality perception result.

8. The multi-modal personality perception device based on multiple associated features and graph relationship attention according to claim 7, characterized in that: Modal feature extraction network module, used for: S31, inputting the visual modality input and the audio modality input into a scene-audio association feature extraction module to obtain a scene-audio association feature; S32, inputting the visual modality input into a scene-description word association feature extraction module to obtain a scene-description word association feature; S33, inputting the audio modal input into an audio-description word association feature extraction module to obtain an audio-description word association feature; S34, inputting the text modality input into the Roberta model to obtain text modality features.

9. A multimodal personality perception device, characterized in that: The multimodal personality perception device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images