Personality prediction method and device fusing CLIP and adaptive graph transformation network
By integrating personality prediction methods of CLIP and adaptive graph transform networks, multimodal correlation characteristics are extracted, and the problem of insufficient accuracy of personality prediction in the prior art is solved, and more efficient personality trait prediction is achieved.
Patent Information
- Application Number
- CN202411145652.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-20
AI Technical Summary
The prior art ignores the exploration of correlation characteristics of different modalities in personality prediction, resulting in the failure to further improve the accuracy of personality prediction.
Using personality prediction method that integrates CLIP and adaptive graph transformation network, audio-scene correlation features are extracted through Wav2CLIP-graph, scene-personality descriptor correlation features are extracted, audio representation features are extracted based on VGGish CNN-RNN model, and multimode The state channel attention residual fusion module forms fusion multimodal characteristics to predict personality traits.
The accuracy of personality prediction is significantly improved, and the multimodal correlation characteristics are explored through the multimodal feature learning framework, which improves the accuracy and robustness of personality analysis.
Smart Images

Figure CN119068393A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of emotional computing technology, and in particular to a personality prediction method and device integrating CLIP and an adaptive graph transformation network. Background Art
[0002] Affective computing is an emerging field that focuses on the interaction between humans and machines, with the goal of creating machines that can understand, interpret, and respond to human emotions. Personality computing is a subfield of affective computing that analyzes, identifies, and synthesizes personality traits. Personality traits are persistent patterns of thinking, feeling, and behavior that shape and distinguish one person's personality from others. Therefore, personality computing combines two scientific fields, psychology and artificial intelligence, making it a relevant research field. With the booming development of multimedia social platforms such as YouTube, Instagram, and TikTok, people are increasingly sharing their lives through videos. In recent years, personality analysis has received increasing attention among researchers in computer vision, linguistics, and other related disciplines. In the past few decades, some researchers have established various personality trait models using psychological scales. For example, the Big Five personality model aims to assess personality through five traits: openness (O), conscientiousness (C), extraversion (E), agreeableness (A), and neuroticism (N). Personality traits can influence people's decisions and preferences, and personality prediction has many practical applications in job interviews, emotional marriage counseling, purchasing behavior, etc.
[0003] From the perspective of actual performance, the existing excellent algorithms mainly consider the singleness of personality prediction datasets, use pre-trained models to fully extract single modality features, and fuse features of different modalities through different feature fusion strategies. Although these pre-trained models have been trained on medium-sized datasets, their ability to extract features still needs to be improved due to the limited number of samples in the pre-training datasets. At the same time, although the personality prediction problem is a multimodal analysis task, these excellent algorithms ignore the exploration of the associated features of different modalities, thus failing to further improve the accuracy of personality prediction. Summary of the invention
[0004] In order to solve the technical problem that the prior art ignores the exploration of different modal correlation features and thus fails to further improve the accuracy of personality prediction, the embodiment of the present invention provides a personality prediction method and device integrating CLIP and adaptive graph transformation network. The technical solution is as follows:
[0005] On the one hand, a personality prediction method integrating CLIP and adaptive graph transformation network is provided, the method is implemented by a personality prediction device, and the method comprises:
[0006] S1. Obtain video data for personality prediction.
[0007] S2. According to the video data and Wav2CLIP-graph method, audio-scene association features are obtained.
[0008] S3. Based on the video data and CLIP method, the scene-personality descriptor association features are obtained.
[0009] S4. Obtain audio representation features based on video data and the VGGish CNN-RNN method.
[0010] S5. Input the audio-scene association features, scene-personality descriptor association features and audio representation features into the multimodal channel attention residual fusion module to obtain fused multimodal features, and then obtain the personality trait prediction results.
[0011] Optionally, the audio-scene association features are obtained according to the video data and the Wav2CLIP-graph method in S2, including:
[0012] S21. For the video data, feature extraction is performed through the pre-trained Wav2CLIP to obtain the audio image feature matrix and the scene image feature matrix of each frame image of the video data.
[0013] S22. Construct an audio sub-block relationship graph according to the audio image feature matrix, and construct a scene relationship graph according to the scene image feature matrix.
[0014] S23. Obtaining the audio sub-block relationship graph aggregation features and the scene relationship graph aggregation features according to the audio sub-block relationship graph, the scene relationship graph, and the adaptive graph conversion network.
[0015] S24, calculating similarity scores for the aggregated features of the audio sub-block relationship graph and the aggregated features of the scene relationship graph to obtain audio-scene association features.
[0016] Optionally, constructing a scene relationship graph according to the scene image feature matrix in S22 includes:
[0017] S221. Use sine function and cosine function to generate a position coding vector for each image feature in the scene image feature matrix, and add the position coding vector to the corresponding scene image feature matrix to obtain a new scene image feature matrix.
[0018] S222. Use the new scene image feature matrix as a node and the cosine similarities between different nodes as edges to construct a scene relationship graph.
[0019] Optionally, the adaptive graph conversion network in S23 is as shown in the following formula (1):
[0020] (1)
[0021] In the formula, represents the generated graph representation vector, Represents the activation function RELU, represents the input feature matrix, represents the batch normalization layer, express The degree matrix of represents the input adjacency matrix, represents the adjacency matrix containing self-loops, represents the weight term of graph convolution, represents the bias term of graph convolution, represents the sum of the fused generated graph representation vector and the multi-head self-attention module, Representation layer normalization layer, Represents the multi-head self-attention module of Transformer, with express, represents the weight term of the first fully connected layer, represents the bias term of the first fully connected layer, represents the weight term of the second fully connected layer, Represents the bias term of the second fully connected layer.
[0022] Optionally, the similarity score calculation process in S24 is as shown in the following formula (2):
[0023] (2)
[0024] In the formula, Represents the similarity score between the aggregated features of the audio sub-block relationship graph and the aggregated features of the scene relationship graph, Represents the aggregation features of the scene relationship graph, Represents the aggregated features of the audio sub-block relationship graph, represents the index of each feature vector, Indicates a preset value.
[0025] Optionally, in S3, scene-personality descriptor association features are obtained based on the video data and the CLIP method, including:
[0026] S31. Generate scene image embedding according to video data and image encoder.
[0027] S32. Build a personality description word list, and generate description embedding according to the personality description word list and the text encoder.
[0028] S33. Perform dot product calculation on the scene image embedding and the description embedding to obtain a similarity score, and obtain scene-personality description word association features based on the similarity score.
[0029] Optionally, in S4, audio representation features are obtained according to the video data and the VGGish CNN-RNN method, including:
[0030] S41. Obtain audio data according to the video data, and obtain a logarithmic Mel-spectrogram according to the audio data.
[0031] S42. Use the pre-trained VGGish CNN as the feature extractor of the audio modality, perform feature extraction on the logarithmic Mel-spectrogram, and obtain the audio modality features.
[0032] S43. Use the RNN model to extract features of the audio modal features to obtain audio representation features.
[0033] On the other hand, a personality prediction device integrating CLIP and an adaptive graph transformation network is provided, and the device is applied to a personality prediction method integrating CLIP and an adaptive graph transformation network, and the device comprises:
[0034] The acquisition module is used to acquire video data to be used for personality prediction.
[0035] The audio-scene correlation feature extraction module is used to obtain audio-scene correlation features based on video data and the Wav2CLIP-graph method.
[0036] The scene-personality descriptor association feature extraction module is used to obtain scene-personality descriptor association features based on video data and the CLIP method.
[0037] The audio representation feature extraction module is used to obtain audio representation features based on video data and the VGGish CNN-RNN method.
[0038] The output module is used to input the audio-scene association features, the scene-personality descriptor association features and the audio representation features into the multimodal channel attention residual fusion module to obtain the fused multimodal features, and then obtain the personality trait prediction results.
[0039] Optionally, the audio-scene association feature extraction module is further used to:
[0040] S21. For the video data, feature extraction is performed through the pre-trained Wav2CLIP to obtain the audio image feature matrix and the scene image feature matrix of each frame image of the video data.
[0041] S22. Construct an audio sub-block relationship graph according to the audio image feature matrix, and construct a scene relationship graph according to the scene image feature matrix.
[0042] S23. Obtaining the audio sub-block relationship graph aggregation features and the scene relationship graph aggregation features according to the audio sub-block relationship graph, the scene relationship graph, and the adaptive graph conversion network.
[0043] S24, calculating similarity scores for the aggregated features of the audio sub-block relationship graph and the aggregated features of the scene relationship graph to obtain audio-scene association features.
[0044] Optionally, the audio-scene association feature extraction module is further used to include:
[0045] S221. Use sine function and cosine function to generate a position coding vector for each image feature in the scene image feature matrix, and add the position coding vector to the corresponding scene image feature matrix to obtain a new scene image feature matrix.
[0046] S222. Use the new scene image feature matrix as a node and the cosine similarities between different nodes as edges to construct a scene relationship graph.
[0047] Optionally, an adaptive graph transformation network is shown in the following formula (1):
[0048] (1)
[0049] In the formula, represents the generated graph representation vector, Represents the activation function RELU, represents the input feature matrix, represents the batch normalization layer, express The degree matrix of represents the input adjacency matrix, represents the adjacency matrix containing self-loops, represents the weight term of graph convolution, represents the bias term of graph convolution, represents the sum of the fused generated graph representation vector and the multi-head self-attention module, Representation layer normalization layer, Represents the multi-head self-attention module of Transformer, with express, represents the weight term of the first fully connected layer, represents the bias term of the first fully connected layer, represents the weight term of the second fully connected layer, Represents the bias term of the second fully connected layer.
[0050] Optionally, the similarity score calculation process is as shown in the following formula (2):
[0051] (2)
[0052] In the formula, Represents the similarity score between the aggregated features of the audio sub-block relationship graph and the aggregated features of the scene relationship graph, Represents the aggregation features of the scene relationship graph, Represents the aggregated features of the audio sub-block relationship graph, represents the index of each feature vector, Indicates a preset value.
[0053] Optionally, the scene-personality description word association feature extraction module is further used to:
[0054] S31. Generate scene image embedding according to video data and image encoder.
[0055] S32. Build a personality description word list, and generate description embedding according to the personality description word list and the text encoder.
[0056] S33. Perform dot product calculation on the scene image embedding and the description embedding to obtain a similarity score, and obtain scene-personality description word association features based on the similarity score.
[0057] Optionally, the audio representation feature extraction module is further used to:
[0058] S41. Obtain audio data according to the video data, and obtain a logarithmic Mel-spectrogram according to the audio data.
[0059] S42. Use the pre-trained VGGish CNN as the feature extractor of the audio modality, perform feature extraction on the logarithmic Mel-spectrogram, and obtain the audio modality features.
[0060] S43. Use the RNN model to extract features of the audio modal features to obtain audio representation features.
[0061] On the other hand, a personality prediction device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the personality prediction methods integrating CLIP and an adaptive graph transformation network is implemented.
[0062] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned personality prediction methods integrating CLIP and adaptive graph transformation network.
[0063] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0064] In an embodiment of the present invention, a multimodal feature learning framework is introduced, which combines CLIP and an adaptive graph conversion network for explicit personality analysis, further explores multimodal correlation features, and improves the accuracy of personality prediction to solve the problems of the prior art. The framework covers visual, audio and text modalities. The present invention proposes a new method (Wav2CLIP-graph) for mining audio-scene correlation features of visual and audio modalities. First, it uses Wav2CLIP to extract scene image and audio image features of each frame from short videos. In addition, the present invention constructs a relationship graph in these two modalities and proposes an adaptive graph conversion network to extract audio and scene features respectively. Finally, the present invention uses similarity scores as audio-scene correlation features. In addition, the present invention uses CLIP to extract correlation features between scene personality descriptors and personality descriptions to improve the accuracy of the framework. First, the text encoder of CLIP is used to generate the embedding of personality descriptors. Second, the image encoder of CLIP is used to create the embedding of scene images. Finally, the dot product is used to calculate the similarity features between the embedding of personality descriptors and the image embedding as scene-description correlation features. Then, two MLPs are used to reduce the dimension of the correlation features. In order to make full use of the complementarity between multiple modal features and enhance the robustness of the framework, the present invention uses VGGish CNN and RNN models to extract audio features. Finally, a multimodal channel attention residual fusion module forms a feature vector as the input of the MLP model for predicting personality traits. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0066] Figure 1 It is a flow chart of a personality prediction method integrating CLIP and adaptive graph transformation network provided by an embodiment of the present invention;
[0067] Figure 2 It is a diagram of the overall structure of personality prediction integrating CLIP and adaptive graph transformation network provided by an embodiment of the present invention;
[0068] Figure 3 It is an audio-scene association feature flow chart based on Wav2CLIP-graph provided by an embodiment of the present invention;
[0069] Figure 4 is a flow chart of scene personality description word association features using pre-trained CLIP provided by an embodiment of the present invention;
[0070] Figure 5 is a flow chart of audio representation feature extraction provided by an embodiment of the present invention;
[0071] Figure 6 is a schematic diagram of a multimodal channel attention residual fusion module provided by an embodiment of the present invention;
[0072] Figure 7 It is a block diagram of a personality prediction device integrating CLIP and adaptive graph transformation network provided by an embodiment of the present invention;
[0073] Figure 8 It is a structural schematic diagram of a personality prediction device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0074] The technical solution of the present invention is described below in conjunction with the accompanying drawings.
[0075] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0076] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.
[0077] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.
[0078] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0079] The embodiment of the present invention provides a personality prediction method integrating CLIP and adaptive graph transformation network, which can be implemented by a personality prediction device, which can be a terminal or a server. Figure 1 The flowchart of the personality prediction method integrating CLIP and adaptive graph transformation network is shown in FIG. The processing flow of the method may include the following steps:
[0080] S1. Obtain video data for personality prediction.
[0081] S2. According to the video data and Wav2CLIP-graph method, audio-scene association features are obtained.
[0082] S3. Based on the video data and CLIP method, the scene-personality descriptor association features are obtained.
[0083] S4. Obtain audio representation features based on video data and the VGGish CNN-RNN method.
[0084] S5. Input the audio-scene association features, scene-personality descriptor association features, and audio representation features into the multimodal channel attention residual fusion module to obtain fused multimodal features, and then obtain personality trait prediction results.
[0085] Automatic personality prediction has become a challenging topic in computer vision. This paper proposes a novel multimodal feature learning framework for analyzing personality traits from short videos. First, the pre-trained Wav2CLIP is used to extract audio and scene features from different frames in the video, and an audio and scene relationship graph is constructed. Then, an adaptive graph transformation network is proposed to extract and aggregate audio and scene features respectively. Finally, the audio-scene association features are obtained by calculating their similarity. In addition, the present invention uses CLIP (Constrastive Language-Image Pre-training) as a feature extractor, and uses personality descriptors and scene embeddings to generate two feature vectors, and the dot product of these two feature vectors is used as the scene-description association feature. In addition, to enhance the robustness of the framework, the present invention uses a pre-trained and fine-tuned VGGish CNN model to extract audio spatial features, and uses RNN (Recurrent Neural Network) to extract significant audio temporal features. Finally, a multimodal channel attention residual fusion module is designed to fuse multimodal features and predict personality traits through an MLP (Multilayer Perceptron) regression model.
[0086] Contrastive Language-Image Pre-training (CLIP) is a new approach to learn image and text representations from natural language supervision. CLIP is trained on a large-scale Internet-collected dataset containing 400 million (image, text) pairs. In general, CLIP consists of two encoders: an image encoder based on ViT (Vision Transformer) and a text encoder based on BERT (Bidirectional Encoder Representations from Transformers). Overall, CLIP pre-training aims to learn a multimodal embedding space where image and text embeddings are semantically similar when they are visually and conceptually related, and semantically dissimilar when they are unrelated. As a result, CLIP has good generalization ability and can be adapted to specific tasks without fine-tuning. It can leverage text as a rich source of supervision to learn visual concepts without explicit annotations. Next, the audio-visual correspondence Wav2CLIP model based on CLIP is proposed to obtain a strong audio representation, projecting the audio into an embedding space shared with images and text. The way Wav2CLIP works is by freezing the CLIP visual model, running it on the visual stream of the video, and training a new audio model to predict accurate CLIP embeddings in the audio stream.
[0087] In this invention, CLIP and Wav2CLIP are used as feature extractors to predict personality, using CLIP's text encoder to create embeddings of personality descriptors for personality traits, Wav2CLIP's image encoder to generate embeddings of visual cues, and its audio encoder to obtain audio representations. The invention assumes that the psychometric information contained in these two large-scale pre-trained models can be exploited to enhance the generalization ability of the models, thereby improving the prediction performance.
[0088] like Figure 2 As shown, the input of the framework of the present invention is a short video and personality descriptors, which mainly includes four components: the first is feature extraction using different modality-specific networks. The second is feature fusion, MLP-based modeling and final prediction. The input video is segmented into visual, textual and audio input streams. For feature extraction, the framework of the present invention uses a separate modality-specific network to learn personality features; then, a multimodal channel attention residual fusion module is used to fuse these learned high-level features. Finally, the fused features are fed into the MLP model to predict the scores of different personality traits.
[0089] Figure 2The proposed framework structure of video-based explicit personality analysis is presented. The framework of the present invention includes three feature extraction modules, namely, audio-scene association feature extraction using Wav2CLIP-graph, scene personality description word association feature extraction using pre-trained CLIP, and audio feature extraction using pre-trained VGGish CNN.
[0090] Specifically, S1. Obtain video data for personality prediction.
[0091] S2. According to the video data and Wav2CLIP-graph method, audio-scene association features are obtained.
[0092] In a feasible implementation, Figure 3 As shown, in order to extract audio scene related features, the present invention first constructs a scene image relationship subgraph and an audio sub-block relationship subgraph, and then proposes an adaptive graph transformation relationship network to aggregate unimodal audio and scene features respectively. Finally, the similarity degree is used to obtain audio scene related features.
[0093] Specifically, the above step S2 may include the following steps S21-S24:
[0094] S21. For the video data, feature extraction is performed through the pre-trained Wav2CLIP to obtain the audio image feature matrix and the scene image feature matrix of each frame image of the video data.
[0095] In a feasible implementation, the scene image feature matrix , the size is (n×768), where (n=15) is the number of frame images from the short video.
[0096] S22. Construct an audio sub-block relationship graph according to the audio image feature matrix, and construct a scene relationship graph according to the scene image feature matrix.
[0097] Optionally, constructing a scene relationship graph according to the scene image feature matrix in S22 may include the following steps S221-S222:
[0098] S221. Use sine function and cosine function to generate a position coding vector for each image feature in the scene image feature matrix, and add the position coding vector to the corresponding scene image feature matrix to obtain a new scene image feature matrix.
[0099] S222. Use the new scene image feature matrix as a node and the cosine similarities between different nodes as edges to construct a scene relationship graph.
[0100] In a feasible implementation manner, the present invention uses sine and cosine functions as the scene image feature matrix Generate position encoding vectors for each image feature in and add them to the original scene image features to obtain a new scene image feature matrix .
[0101] Furthermore, to construct the scene-image relationship subgraph , where the node Aims to capture the semantics of different frames, edge Used to measure the relationship between different nodes. These visual features As a node of the scene-image relationship subgraph ,in and Respectively represent The visual features of the frame image and the dimensions of the visual features. It is determined by calculating the cosine similarity between different nodes in the scene-image relationship subgraph. Each node uses the cosine similarity based The nearest neighbor algorithm is used to establish edge connections. The calculation formula is as follows:
[0102] (1)
[0103] here, , where the cosine similarity Indicates and Cosine similarity between nodes.
[0104] Optionally, constructing the audio sub-block relationship diagram according to the audio image feature matrix in S22 may be: first, use the ffmpeg tool to convert the video into audio data for extracting audio features. After a series of operations such as framing, pre-emphasis, window function, fast Fourier transform, Mel filtering and logarithmic transform, the audio is converted into a logarithmic Mel spectrum image. Next, first divide the logarithmic Mel spectrum image into several sub-blocks, and each sub-block is separately input into the audio encoder of Wav2CLIP to extract the sub-block features, thereby obtaining the sub-block feature matrix (m×768), where (m=15) represents the number of sub-blocks extracted from the short video. Similar to building a scene image relationship sub-graph, an audio relationship sub-graph can also be built. , where the node Aims to capture the semantics of different sub-blocks, while the edges is used to measure the relationship between other nodes. It can extract the semantic relationship between different sub-blocks in the audio modality for personality prediction. For each node, select The nearest neighbor algorithm is used to establish edge connections. Here, .
[0105] S23. Obtaining the audio sub-block relationship graph aggregation features and the scene relationship graph aggregation features according to the audio sub-block relationship graph, the scene relationship graph, and the adaptive graph conversion network.
[0106] In a feasible implementation, the present invention is inspired by the idea of Transformer. The present invention constructs an adaptive graph transformation network, including a graph convolution layer, a multi-head self-attention module ( ), feedforward neural network and residual structure, which can be described as follows:
[0107] (2)
[0108] in, is the activation function RELU, is the input scene image feature matrix or audio sub-block feature matrix, is an adjacency matrix containing self-loops, yes The degree matrix of represents the input adjacency matrix, i.e., a diagonal matrix, is a batch normalization layer, and are the weight and bias of graph convolution respectively, is the layer normalization layer, is the multi-head self-attention module of Transformer, and and are the weights and bias terms of the two fully connected layers.
[0109] S24, calculating similarity scores for the aggregated features of the audio sub-block relationship graph and the aggregated features of the scene relationship graph to obtain audio-scene association features.
[0110] In a feasible implementation, when constructing a scene image relationship subgraph and audio relationship subgraph After that, the two relationship graphs are first subjected to the maximum pooling operation to obtain the aggregated features of the scene image relationship subgraphs. Aggregate features of audio-related subgraphs Finally, the scene-audio correlation features are extracted from the two subgraph aggregate features by using cosine similarity, which exploits the intramodal and cross-modal properties of knowledge for personality prediction tasks. First, the scene-image relation subgraph aggregate features are extracted After feeding the new scene image features into two consecutive graph encoders, the features of different scene images are obtained. (15×768). Then, a fully connected layer is used to transform Mapped to dim=128 dimension, use global maximum pooling to get the final feature representation of the entire scene image relationship graph (1×128), described as follows:
[0111] (3)
[0112] in, and are the weights and biases of the feature map layer, Represents a global maximum pooling operation.
[0113] Finally, the present invention calculates bit by bit and similarity features to obtain the audio-scene correlation feature representation of the scene image and audio sub-block (1×128). The entire calculation process is as follows:
[0114] (4)
[0115] In the formula, represents the index of each feature vector, Indicates a minimum value of 10-8 to be added to prevent the dividend from being 0.
[0116] S3. Based on the video data and CLIP method, the scene-personality descriptor association features are obtained.
[0117] like Figure 4 As shown in the figure, the present invention designs a scene personality description word associated feature extraction method based on CLIP. The method includes three main steps: 1) using CLIP's image encoder to generate embeddings of visual cues; 2) using CLIP's text encoder to generate embeddings of personality description words respectively; 3) calculating the similarity score between the scene image embedding and the description word embedding by calculating the dot product between them, and reducing the dimension of the feature through a multi-layer perceptron.
[0118] Optionally, the above step S3 may include the following steps S31-S33:
[0119] S31. Generate scene image embedding according to video data and image encoder.
[0120] In one possible implementation, the present invention uses the text encoder of CLIP to generate description embeddings for these personality descriptors. At the same time, the "openai / clip-vit-large-patch14-336" model based on the ViT architecture is used as the image encoder of CLIP to generate embeddings for images from videos.
[0121] S32. Build a personality description word list, and generate description embedding according to the personality description word list and the text encoder.
[0122] In one possible implementation, the present invention generates embeddings for personality descriptors using a list of 1,608 personality adjectives that accurately describe human personality traits. The list contains personality terms carefully selected from personality surveys and multiple dictionaries to ensure the diversity and representativeness of the descriptions to effectively capture the complexity of human personality.
[0123] S33. Perform dot product calculation on the scene image embedding and the description embedding to obtain a similarity score, and obtain scene-personality description word association features based on the similarity score.
[0124] In one possible implementation, using the embedding of images and descriptions, the present invention calculates the similarity scores between them through dot product operations and uses them as feature vectors. This results in a feature vector containing 1608 values for each frame image, each value corresponding to the similarity score between the frame image and the corresponding personality descriptor. Then, the feature vectors of all frame images in each video are averaged to obtain a scene personality descriptor association feature containing 1608 features.
[0125] Specifically, assuming express A collection of personality descriptors. Extracted from the video For each personality description word , generate text embeddings using CLIP’s text encoder At the same time, for each frame image , using CLIP’s image encoder to generate image embeddings Then, we calculate a feature matrix , where each entry Indicates Frame image and The similarity score between the personality descriptors. This score is calculated as the corresponding Personality descriptors and Dot product of frame image embedding:
[0126] (5)
[0127] Furthermore, the pre-trained CLIP model learns a joint embedding space to map scene images and personality descriptors into the same space. The present invention averages the feature matrices of all image frames in the video:
[0128] (6)
[0129] in, It is The feature vector of the video, It is The number of frames in a video, the present invention sets The present invention constructs and trains an MLP regression model, which includes two fully connected (FC) layers to reduce the dimension of the above scene personality descriptor associated features. Each FC layer includes a linear layer, a batch normalization (BN) layer, a ReLU activation layer and a Dropout layer (parameter p=0.3), which reduces the dimension of the feature from 1608 to 512, and finally to 256.
[0130] S4. Obtain audio representation features based on video data and the VGGish CNN-RNN method.
[0131] Optionally, the above step S4 may include the following steps S41-S43:
[0132] S41. Obtain audio data according to the video data, and obtain a logarithmic Mel-spectrogram according to the audio data.
[0133] In one possible implementation, converting audio to Log-Mel spectrum features is a key step in the process of the present invention. This feature is processed by Python's resample and soundfile libraries, which are very important for reading and writing audio. The input size of the Log-Mel spectrum is changed to 96×64 to adapt to the VGGish CNN model. It should be noted that audio can reflect a person's personality characteristics through speaking style and intonation changes.
[0134] S42. Use the pre-trained VGGish CNN as the feature extractor of the audio modality, perform feature extraction on the logarithmic Mel-spectrogram, and obtain the audio modality features.
[0135] In a feasible implementation, the present invention uses a pre-trained VGGish CNN as a feature extractor for the audio modality. Here, the pre-trained VGGish CNN model is structurally modified based on VGG 11, removing the last set of convolutional layers and maximum pooling layers, and changing the last fully connected layer from 1000 dimensions to 128 dimensions.
[0136] Specifically, the VGGish CNN model converts each 15-second audio signal into a 128-dimensional high-level feature vector with semantics and meaning. Therefore, for a video of about 15 seconds, the potential feature vector is [25×128].
[0137] S43. Use the RNN model to extract features of the audio modal features to obtain audio representation features.
[0138] In a feasible implementation, the RNN model is used to extract the temporal features of each audio as the final audio representation features, such as Figure 5 shown.
[0139] S5. Input the audio-scene association features, scene-personality descriptor association features, and audio representation features into the multimodal channel attention residual fusion module to obtain fused multimodal features, and then obtain personality trait prediction results.
[0140] In a feasible implementation, the existing method directly connects multiple feature vectors, ignoring the fact that different features have different effects on personality prediction. Therefore, the present invention designs a multimodal channel attention residual fusion module.
[0141] Specifically, the three feature vectors obtained are concatenated into a fused feature vector, which is then used as the input of the module to evaluate the contribution of each branch and reduce the information redundancy caused by feature diversity. A residual structure is introduced in this module to reduce information loss. Figure 6 As shown in Figure 2, the module includes two fully connected layers to calculate the attention weights for each dimension in the multimodal representation F. , It can be calculated as:
[0142] (7)
[0143] in, , , b and c represent the weight matrix and bias term of the two fully connected layers respectively. The tanh function is used to limit the attention weight to the interval [-1, 1], and then the attention vector With multimodal vector Element-wise multiplication is performed on each dimension of . The formula is as follows:
[0144] (8)
[0145] here, is the output of the module, including a residual structure, which is used to predict the five personality traits through MLP.
[0146] Furthermore, in the present invention, a three-layer perceptron (MLP) is a Big Five personality prediction model. For this prediction model, the RELU activation function is used when activating the first two linear layers, and a dropout layer is added after the activation. The last layer uses a sigmoid activation function to map the output predicted by the model to the range [0,1].
[0147] The present invention uses a multi-task loss function to jointly optimize the entire framework, which can be defined as follows:
[0148] (9)
[0149] in, is the root mean square error loss function, yes Loss function, yes Loss function.
[0150] The Big Five personality prediction system based on multimodal and graph structure feature learning network disclosed in the present invention can predict people's Big Five personality traits. The system can be widely used in crime detection, medical rehabilitation, talent recruitment and other fields.
[0151] This paper proposes a multimodal feature learning framework that combines CLIP and an adaptive graph transformation network to capture multimodal features for predictive personality analysis.
[0152] The present invention proposes a novel Wav2CLIP-graph method for constructing an audio and scene relationship graph. The present invention designs an adaptive graph conversion network and defines similarity features to obtain audio-scene association features for personality analysis.
[0153] Based on personality descriptors, the present invention can use pre-trained CLIP to obtain scene-personality descriptor association features. This feature can significantly improve the accuracy of personality prediction.
[0154] In an embodiment of the present invention, a multimodal feature learning framework is introduced, which combines CLIP and an adaptive graph conversion network for explicit personality analysis, further explores multimodal correlation features, and improves the accuracy of personality prediction to solve the problems of the prior art. The framework covers visual, audio and text modalities. The present invention proposes a new method (Wav2CLIP-graph) for mining audio-scene correlation features of visual and audio modalities. First, it uses Wav2CLIP to extract scene image and audio image features of each frame from a short video. In addition, the present invention constructs a relationship graph in these two modalities and proposes an adaptive graph conversion network to extract audio and scene features respectively. Finally, the present invention uses similarity scores as audio-scene correlation features. In addition, the present invention uses CLIP to extract correlation features between scene personality descriptors and personality descriptions to improve the accuracy of the framework. First, the text encoder of CLIP is used to generate the embedding of personality descriptors. Second, the image encoder of CLIP is used to create the embedding of scene images. Finally, the dot product is used to calculate the similarity features between the embedding of personality descriptors and the image embedding as scene-description correlation features. Then, two MLPs are used to reduce the dimension of the correlation features. In order to make full use of the complementarity between multiple modal features and enhance the robustness of the framework, the present invention uses VGGish CNN and RNN models to extract audio features. Finally, a multimodal channel attention residual fusion module forms a feature vector as the input of the MLP model for predicting personality traits.
[0155] Figure 7 : is a block diagram of a personality prediction device integrating CLIP and adaptive graph transformation network according to an exemplary embodiment, wherein the device is used for a personality prediction method integrating CLIP and adaptive graph transformation network. Figure 7 The device includes an acquisition module 310, an audio-scene associated feature extraction module 320, a scene-personality description word associated feature extraction module 330, an audio representation feature extraction module 340 and an output module 350. Among them:
[0156] The acquisition module 310 is used to acquire video data for personality prediction.
[0157] The audio-scene correlation feature extraction module 320 is used to obtain audio-scene correlation features according to the video data and the Wav2CLIP-graph method.
[0158] The scene-personality descriptor association feature extraction module 330 is used to obtain scene-personality descriptor association features according to the video data and the CLIP method.
[0159] The audio representation feature extraction module 340 is used to obtain audio representation features according to the video data and the VGGish CNN-RNN method.
[0160] The output module 350 is used to input the audio-scene association features, the scene-personality descriptor association features and the audio representation features into the multimodal channel attention residual fusion module to obtain fused multimodal features, and then obtain personality trait prediction results.
[0161] In an embodiment of the present invention, a multimodal feature learning framework is introduced, which combines CLIP and an adaptive graph conversion network for explicit personality analysis, further explores multimodal correlation features, and improves the accuracy of personality prediction to solve the problems of the prior art. The framework covers visual, audio and text modalities. The present invention proposes a new method (Wav2CLIP-graph) for mining audio-scene correlation features of visual and audio modalities. First, it uses Wav2CLIP to extract scene image and audio image features of each frame from a short video. In addition, the present invention constructs a relationship graph in these two modalities and proposes an adaptive graph conversion network to extract audio and scene features respectively. Finally, the present invention uses similarity scores as audio-scene correlation features. In addition, the present invention uses CLIP to extract correlation features between scene personality descriptors and personality descriptions to improve the accuracy of the framework. First, the text encoder of CLIP is used to generate the embedding of personality descriptors. Second, the image encoder of CLIP is used to create the embedding of scene images. Finally, the dot product is used to calculate the similarity features between the embedding of personality descriptors and the image embedding as scene-description correlation features. Then, two MLPs are used to reduce the dimension of the correlation features. In order to make full use of the complementarity between multiple modal features and enhance the robustness of the framework, the present invention uses VGGish CNN and RNN models to extract audio features. Finally, a multimodal channel attention residual fusion module forms a feature vector as the input of the MLP model for predicting personality traits.
[0162] Figure 8 is a schematic diagram of the structure of a personality prediction device provided by an embodiment of the present invention, such as Figure 8 As shown, the personality prediction device may include the above Figure 7 The personality prediction device 410 shown integrates CLIP and adaptive graph transformation network. Optionally, the personality prediction device 410 may include a first processor 2001.
[0163] Optionally, the personality prediction device 410 may further include a memory 2002 and a transceiver 2003 .
[0164] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.
[0165] Combine the following Figure 8 The components of the personality prediction device 410 are specifically introduced as follows:
[0166] The first processor 2001 is the control center of the personality prediction device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).
[0167] Optionally, the first processor 2001 may perform various functions of the personality prediction device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .
[0168] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.
[0169] In a specific implementation, as an embodiment, the personality prediction device 410 may also include multiple processors, such as Figure 8 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0170] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.
[0171] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and access the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0172] The transceiver 2003 is used to communicate with a network device or a terminal device.
[0173] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.
[0174] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the personality prediction device 410 through an interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0175] It should be noted that Figure 8 The structure of the personality prediction device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0176] In addition, the technical effects of the personality prediction device 410 can refer to the technical effects of the personality prediction method integrating CLIP and adaptive graph transformation network described in the above method embodiment, which will not be repeated here.
[0177] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0178] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0179] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0180] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.
[0181] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0182] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0183] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0185] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0186] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0187] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0188] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.
[0189] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A personality prediction method integrating CLIP and adaptive graph transformation network, characterized in that: The method comprises: S1. Obtaining video data for personality prediction; S2. Obtaining audio-scene association features according to the video data and the Wav2CLIP-graph method; S3, obtaining scene-personality descriptor association features according to the video data and the CLIP method; S4. Obtaining audio representation features according to the video data and the VGGish CNN-RNN method; S5. Input the audio-scene association features, scene-personality descriptor association features and audio representation features into a multimodal channel attention residual fusion module to obtain fused multimodal features, and then obtain personality trait prediction results.
2. The personality prediction method integrating CLIP and adaptive graph transformation network according to claim 1, characterized in that: The step of obtaining audio-scene association features according to the video data and the Wav2CLIP-graph method in S2 includes: S21. For the video data, extract features by using the pre-trained Wav2CLIP to obtain an audio image feature matrix and a scene image feature matrix for each frame of the video data; S22, constructing an audio sub-block relationship graph according to the audio image feature matrix, and constructing a scene relationship graph according to the scene image feature matrix; S23, obtaining an audio sub-block relationship graph aggregation feature and a scene relationship graph aggregation feature according to the audio sub-block relationship graph, the scene relationship graph, and the adaptive graph conversion network; S24, calculating similarity scores for the aggregated features of the audio sub-block relationship graph and the aggregated features of the scene relationship graph to obtain audio-scene association features.
3. The personality prediction method integrating CLIP and adaptive graph transformation network according to claim 2, characterized in that: The step of constructing a scene relationship graph according to the scene image feature matrix in S22 includes: S221, using a sine function and a cosine function to generate a position coding vector for each image feature in the scene image feature matrix, and adding the position coding vector to the corresponding scene image feature matrix to obtain a new scene image feature matrix; S222: Use the new scene image feature matrix as a node and the cosine similarities between different nodes as edges to construct a scene relationship graph.
4. The personality prediction method integrating CLIP and adaptive graph transformation network according to claim 2, characterized in that: The adaptive graph conversion network in S23 is shown in the following formula (1): (1) In the formula, represents the generated graph representation vector, Represents the activation function RELU, represents the input feature matrix, represents the batch normalization layer, express The degree matrix of represents the input adjacency matrix, represents the adjacency matrix containing self-loops, represents the weight term of graph convolution, represents the bias term of graph convolution, represents the sum of the fused generated graph representation vector and the multi-head self-attention module, Representation layer normalization layer, Represents the multi-head self-attention module of Transformer, with express, represents the weight term of the first fully connected layer, represents the bias term of the first fully connected layer, represents the weight term of the second fully connected layer, Represents the bias term of the second fully connected layer.
5. The personality prediction method integrating CLIP and adaptive graph transformation network according to claim 2, characterized in that: The similarity score calculation process in S24 is shown in the following formula (2): (2) In the formula, Represents the similarity score between the aggregated features of the audio sub-block relationship graph and the aggregated features of the scene relationship graph, Represents the aggregation features of the scene relationship graph, Represents the aggregated features of the audio sub-block relationship graph, represents the index of each feature vector, Indicates a preset value.
6. The personality prediction method integrating CLIP and adaptive graph transformation network according to claim 1, characterized in that: The step S3 obtains scene-personality descriptor association features according to the video data and the CLIP method, including: S31, generating a scene image embedding according to the video data and the image encoder; S32, constructing a personality description word list, and generating a description embedding according to the personality description word list and a text encoder; S33, performing dot product calculation on the scene image embedding and the description embedding to obtain a similarity score, and obtaining a scene-personality description word association feature according to the similarity score.
7. The personality prediction method integrating CLIP and adaptive graph transformation network according to claim 1, characterized in that: The step S4 of obtaining audio representation features according to the video data and the VGGish CNN-RNN method includes: S41, obtaining audio data according to the video data, and obtaining a logarithmic Mel-frequency spectrum according to the audio data; S42, using a pre-trained VGGish CNN as a feature extractor of the audio modality, performing feature extraction on the logarithmic Mel-spectrogram to obtain audio modality features; S43. Use the RNN model to extract features of the audio modal features to obtain audio representation features.
8. A personality prediction device integrating CLIP and an adaptive graph transformation network, wherein the personality prediction device integrating CLIP and an adaptive graph transformation network is used to implement the personality prediction method integrating CLIP and an adaptive graph transformation network as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, used to acquire video data to be used for personality prediction; An audio-scene correlation feature extraction module, used to obtain audio-scene correlation features according to the video data and the Wav2CLIP-graph method; A scene-personality descriptor association feature extraction module, used to obtain scene-personality descriptor association features according to the video data and the CLIP method; An audio representation feature extraction module, used to obtain audio representation features according to the video data and the VGGish CNN-RNN method; The output module is used to input the audio-scene association features, scene-personality descriptor association features and audio representation features into the multimodal channel attention residual fusion module to obtain fused multimodal features, and then obtain personality trait prediction results.
9. A personality prediction device, characterized in that: The personality prediction device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Large five-personality feature prediction method based on multi-modal fusion
CN117592014A
Multi-modal personality prediction method and system fused with space-time diagram attention network
CN118228194A
Method and device for detecting affective events in a video
EP2509006A1
Cited By
Multi-modal personality prediction method based on pre-training model and pyramid graph fusion
CN119337325A
Multi-modal personality prediction method and device based on pre-training and graph relation transformation network
CN119622638A
Motion feature vector optimization method, device and equipment based on multi-modal fusion
CN121214534A
Self-adaptive condition enhancement method and device for face generation, equipment and storage medium
CN121661435A
