Multi-modal personality prediction method and device based on pre-training and graph relation transformation network

Abstract

The invention provides a network multi-modal personality prediction method and device based on pre-training and graph relation transformation, and relates to the technical field of emotion calculation. The method comprises the following steps: acquiring an input video, inputting the input video into a preprocessing module to obtain visual modal information data, text modal information data and audio modal information data, respectively inputting the visual modal information data, the text modal information data and the audio modal information data into a feature extraction module to obtain visual sequence features, audio sequence features and text sequence features, and respectively inputting the visual sequence features, the audio sequence features and the text sequence features into a graph relation transformation network; the method comprises the following steps: acquiring a visual feature, an audio feature and a text feature, inputting the visual feature, the visual-audio associated feature and the visual-text associated feature into a multi-modal attention feature fusion module to obtain a fusion feature, and inputting the fusion feature into a prediction head module to obtain a multi-modal personality prediction result. The invention provides a dynamic personality prediction method suitable for learning and fusing intra-modal and inter-modal features.

Description

Technical Field

[0001] The present invention relates to the field of emotional computing technology, and in particular to a multimodal personality prediction method and device based on pre-training and graph relation transformation network. Background Art

[0002] In the field of affective computing, personality analysis studies how machines can automatically identify or synthesize human personality traits. One of the most widely accepted theories of personality traits is the "Big Five Personality Model". The Big Five Personality Model refers to: openness (O), conscientiousness (C), extraversion (E), agreeableness (A) and neuroticism (N). Automatically detecting a person's personality traits is of vital importance in many practical applications, such as job interviews, recommendation methods, social network analysis, personal assistants, purchasing behavior, healthcare, criminology, and so on.

[0003] From the perspective of multimodal fusion, the inherent heterogeneity between different modalities often introduces information redundancy and distribution gaps, which increases the difficulty of multimodal feature fusion. In this case, most excellent algorithms process the representation of each modality in a holistic learning manner, resulting in the learned multimodal features that may not be refined enough and have redundancy. Recently, pre-training has developed rapidly in multimodal processing, especially for multimodal pre-training models such as CLIP (Contrastive Language Image Pre-training) and Wav2CLIP, which can not only generalize tasks well without fine-tuning, but also provide embeddings corresponding to category labels, making them very suitable as feature extractors for personality prediction tasks that lack data. Summary of the invention

[0004] In order to solve the problem existing in the prior art, although a variety of methods have been proposed for multimodal fusion, they extract and fuse single-modal features and ignore multimodal correlation features, thus failing to further improve the accuracy of personality prediction. At the same time, previous studies mostly use RNN (Recurrent Neural Network) and its variants to learn sequence dependencies. However, they have gradient vanishing / explosion problems and limited ability to learn long-term dependencies. The above problems increase the technical problem of the difficulty of learning dynamic information within the modality. The embodiment of the present invention provides a multimodal personality prediction method and device based on pre-training and graph relational transformation network. The method combines a pre-training model and a graph relational transformation network to analyze personality traits, and on this basis further explores multimodal correlation features and improves the accuracy of personality prediction to solve the above problems. The technical solution is as follows:

[0005] On the one hand, a multimodal personality prediction method based on pre-training and graph relation transformation network is provided, the method is implemented by a multimodal personality prediction device, and the method includes:

[0006] S1. Get input video.

[0007] S2. Input the input video to a preprocessing module to obtain visual modality information data, text modality information data, and audio modality information data.

[0008] S3. Input the visual modal information data, the textual modal information data and the audio modal information data into the feature extraction module respectively to obtain visual sequence features, audio sequence features and textual sequence features.

[0009] S4. Input the visual sequence features, audio sequence features and text sequence features into the graph relation transformation network respectively to obtain visual features, audio features and text features, obtain visual-audio association features based on the visual features and audio features, and obtain visual-text association features based on the visual features and text features.

[0010] S5. Input the visual features, visual-audio association features, and visual-text association features into the multimodal attention feature fusion module to obtain fused features.

[0011] S6. Input the fused features into the prediction head module to obtain the multimodal personality prediction results.

[0012] Optionally, the feature extraction module in S3 includes a visual encoder, an audio encoder, and a text encoder.

[0013] In S3, the visual modality information data, the text modality information data, and the audio modality information data are respectively input into the feature extraction module to obtain visual sequence features, audio sequence features, and text sequence features, including:

[0014] S31. Input the visual modality information data into the visual encoder to obtain visual sequence features; wherein the visual encoder is a CLIP visual encoder.

[0015] S32. Input the audio modal information data into an audio encoder to obtain audio sequence features; wherein the audio encoder is a Wav2CLIP audio encoder.

[0016] S33. Input the text modal information data into a text encoder to obtain text sequence features; wherein the text encoder is a CLIP text encoder.

[0017] Optionally, the step of inputting the visual modality information data into a visual encoder in S31 to obtain visual sequence features includes:

[0018] S311, dividing the visual modal information data into equal intervals to obtain divided visual modal information data.

[0019] S312. For each frame of scene image in the divided visual modality information data, a spatial scene feature sequence is obtained through the CLIP visual encoder.

[0020] S313. For the spatial scene feature sequence, extract sequence information through xLSTM to obtain spatiotemporal scene features.

[0021] S314. For the spatiotemporal scene features, visual sequence features are obtained through global average pooling and a 1-layer MLP network.

[0022] Optionally, in S4, the visual sequence features, the audio sequence features, and the text sequence features are respectively input into the graph relation transformation network to obtain the visual features, the audio features, and the text features, including:

[0023] S41. Obtain position codes for the visual sequence features, audio sequence features and text sequence features respectively according to sine and cosine, and embed the position codes into the corresponding visual sequence features, audio sequence features and text sequence features to obtain visual sequence features after position code embedding, audio sequence features after position code embedding and text sequence features after position code embedding.

[0024] S42. Each time series in the visual sequence features after position coding embedding, the audio sequence features after position coding embedding, and the text sequence features after position coding embedding is regarded as a spatial node, and the cosine similarity function is used to calculate the spatial feature similarity between the spatial nodes.

[0025] S43. According to the spatial feature similarity between spatial nodes, an adjacency matrix is ​​constructed using the k-NN algorithm.

[0026] S44. The adjacency matrix is ​​transmitted and feature information is aggregated through a one-layer graph convolutional network to obtain visual output features, audio output features, and text output features.

[0027] The calculation formula of the graph convolutional network is shown in the following formula (1):

[0028] (1)

[0029] In the formula, Represents the output features after passing through a 1-layer graph convolutional network and batch normalization operation, represents a batch normalization layer, Degree matrix The diagonal matrix of Represents the adjacency matrix with self-loop I added , represents the input features, represents the weight term of the graph convolution of modality s, represents the bias term of the graph convolution of modality s, represents the output features of mode s, represents the ReLU activation function, represents a time series, Represents the feature dimension of each frame.

[0030] S45. Obtain visual features, audio features, and text features according to the visual output features, audio output features, text output features, and a 1-layer transform encoder.

[0031] The calculation formula of the transform encoder is as shown in the following formula (2):

[0032] (2)

[0033] In the formula, Represents space-time characteristics, represents the global average pooling layer, express, represents the weight term of the fully connected layer, Represents the bias term of the fully connected layer.

[0034] Optionally, obtaining visual-audio correlation features according to the visual features and the audio features in S4 includes:

[0035] Through the improved cosine similarity function, the similarity scores of visual features and audio features are calculated bit by bit, and the visual-audio association features are obtained according to the similarity scores.

[0036] Among them, the improved cosine similarity function is shown in the following formula (3):

[0037] (3)

[0038] In the formula, Represents the similarity score of visual features and audio features, Represents visual features, Represents audio features, .

[0039] Through the improved cosine similarity function, the similarity scores of visual features and text features are calculated bit by bit, and the visual-text association features are obtained according to the similarity scores. .

[0040] Optionally, in S5, the visual features, the visual-audio associated features, and the visual-text associated features are input into a multimodal attention feature fusion module to obtain fused features, including:

[0041] S51, respectively obtain the contribution value of the visual feature, the contribution value of the visual-audio associated feature, and the contribution value of the visual-text associated feature through the multimodal attention feature fusion module, as shown in the following formula (4):

[0042] (4)

[0043] In the formula, Indicates characteristics, Indicates stacking operation, represents the visual-audio association feature, Represents visual features, represents the visual-text association feature, represents the global average pooling output, represents global average pooling, represents the contribution value, represents the softmax activation function, represents the weight term of the first linear layer, represents the bias term of the first linear layer, represents the weight term of the second linear layer, Represents the bias term of the second linear layer.

[0044] S52, according to the contribution value and the visual feature, the visual-audio association feature and the visual-text association feature, a fusion feature is obtained, as shown in the following formula (5):

[0045] (5)

[0046] In the formula, Indicates The output of the head, represents the weighted sum operation, represents the fusion feature, Represents a concatenation operation.

[0047] Optionally, the method further includes a multimodal loss module, as shown in the following formula (6):

[0048] (6)

[0049] In the formula, represents the multi-task loss function of the multimodal loss module, represents the root mean square error loss function, represents the Log Cosh loss function, represents the Bell loss function, Represents the Gaussian loss function.

[0050] On the other hand, a multimodal personality prediction device based on pre-training and graph relation transformation network is provided, and the device is applied to a multimodal personality prediction method based on pre-training and graph relation transformation network, and the device comprises:

[0051] The acquisition module is used to obtain input video.

[0052] The preprocessing module is used to input the input video into the preprocessing module to obtain visual modality information data, text modality information data and audio modality information data.

[0053] The feature extraction module is used to input the visual modality information data, the text modality information data and the audio modality information data into the feature extraction module respectively to obtain visual sequence features, audio sequence features and text sequence features.

[0054] The graph relation transformation module is used to input the visual sequence features, audio sequence features and text sequence features into the graph relation transformation network respectively to obtain visual features, audio features and text features, obtain visual-audio association features based on the visual features and audio features, and obtain visual-text association features based on the visual features and text features.

[0055] The multimodal attention feature fusion module is used to input visual features, visual-audio association features and visual-text association features into the multimodal attention feature fusion module to obtain fused features.

[0056] The prediction head module is used to input the fusion features into the prediction head module to obtain the multimodal personality prediction results.

[0057] Optionally, the feature extraction module includes a visual encoder, an audio encoder, and a text encoder.

[0058] The feature extraction module is further used to:

[0059] S31. Input the visual modality information data into the visual encoder to obtain visual sequence features; wherein the visual encoder is a CLIP visual encoder.

[0060] S32. Input the audio modal information data into an audio encoder to obtain audio sequence features; wherein the audio encoder is a Wav2CLIP audio encoder.

[0061] S33. Input the text modal information data into a text encoder to obtain text sequence features; wherein the text encoder is a CLIP text encoder.

[0062] Optionally, the feature extraction module is further used to:

[0063] S311, dividing the visual modal information data into equal intervals to obtain divided visual modal information data.

[0064] S312. For each frame of scene image in the divided visual modality information data, a spatial scene feature sequence is obtained through the CLIP visual encoder.

[0065] S313. For the spatial scene feature sequence, extract sequence information through xLSTM to obtain spatiotemporal scene features.

[0066] S314. For the spatiotemporal scene features, visual sequence features are obtained through global average pooling and a 1-layer MLP network.

[0067] Optionally, the graph relationship transformation module is further used to:

[0068] S41. Obtain position codes for the visual sequence features, audio sequence features and text sequence features respectively according to sine and cosine, and embed the position codes into the corresponding visual sequence features, audio sequence features and text sequence features to obtain visual sequence features after position code embedding, audio sequence features after position code embedding and text sequence features after position code embedding.

[0069] S42. Each time series in the visual sequence features after position coding embedding, the audio sequence features after position coding embedding, and the text sequence features after position coding embedding is regarded as a spatial node, and the cosine similarity function is used to calculate the spatial feature similarity between the spatial nodes.

[0070] S43. According to the spatial feature similarity between spatial nodes, an adjacency matrix is ​​constructed using the k-NN algorithm.

[0071] S44. The adjacency matrix is ​​transmitted and feature information is aggregated through a one-layer graph convolutional network to obtain visual output features, audio output features, and text output features.

[0072] The calculation formula of the graph convolutional network is shown in the following formula (1):

[0073] (1)

[0074] In the formula, Represents the output features after passing through a 1-layer graph convolutional network and batch normalization operation, represents a batch normalization layer, Degree matrix The diagonal matrix of Represents the adjacency matrix with self-loop I added , represents the input features, represents the weight term of the graph convolution of modality s, represents the bias term of the graph convolution of modality s, represents the output features of mode s, represents the ReLU activation function, represents a time series, Represents the feature dimension of each frame.

[0075] S45. Obtain visual features, audio features, and text features according to the visual output features, audio output features, text output features, and a 1-layer transform encoder.

[0076] The calculation formula of the transform encoder is as shown in the following formula (2):

[0077] (2)

[0078] In the formula, Represents space-time characteristics, represents the global average pooling layer, express, represents the weight term of the fully connected layer, Represents the bias term of the fully connected layer.

[0079] Optionally, the graph relationship transformation module is further used to:

[0080] Through the improved cosine similarity function, the similarity scores of visual features and audio features are calculated bit by bit, and the visual-audio association features are obtained according to the similarity scores.

[0081] Among them, the improved cosine similarity function is shown in the following formula (3):

[0082] (3)

[0083] In the formula, Represents the similarity score of visual features and audio features, Represents visual features, Represents audio features, .

[0084] Through the improved cosine similarity function, the similarity scores of visual features and text features are calculated bit by bit, and the visual-text association features are obtained according to the similarity scores. .

[0085] Optionally, the multimodal attention feature fusion module is further used to:

[0086] S51, respectively obtain the contribution value of the visual feature, the contribution value of the visual-audio associated feature, and the contribution value of the visual-text associated feature through the multimodal attention feature fusion module, as shown in the following formula (4):

[0087] (4)

[0088] In the formula, Indicates characteristics, Indicates stacking operation, represents the visual-audio association feature, Represents visual features, represents the visual-text association feature, represents the global average pooling output, represents global average pooling, represents the contribution value, represents the softmax activation function, represents the weight term of the first linear layer, represents the bias term of the first linear layer, represents the weight term of the second linear layer, Represents the bias term of the second linear layer.

[0089] S52, according to the contribution value and the visual feature, the visual-audio association feature and the visual-text association feature, a fusion feature is obtained, as shown in the following formula (5):

[0090] (5)

[0091] In the formula, Indicates The output of the head, represents the weighted sum operation, represents the fusion feature, Represents a concatenation operation.

[0092] Optionally, a multimodal loss module is also included, as shown in the following formula (6):

[0093] (6)

[0094] In the formula, represents the multi-task loss function of the multimodal loss module, represents the root mean square error loss function, represents the Log Cosh loss function, represents the Bell loss function, Represents the Gaussian loss function.

[0095] On the other hand, a multimodal personality prediction device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned multimodal personality prediction methods based on pre-training and graph relation transformation network is implemented.

[0096] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned pre-training and graph relation transformation network based multimodal personality prediction methods.

[0097] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0098] In the present invention, multimodal personality analysis aims to identify the personality traits of people in short videos. Based on the fact that the visual modality plays a dominant role in this task, the present invention proposes a dynamic personality prediction method adapted to learning and fusing intra-modal and inter-modal features. The method of the present invention combines a unimodal visual scene feature and two bimodal association features in a multimodal learning method, which can model spatial and temporal feature relationships. First, the present invention uses an encoder of a pre-trained model to extract unimodal spatial scene features from a video. Then, xLSTM is used to capture long-range sequence dependencies between different scene frames, which are used as scene features. The encoders of CLIP and Wav2CLIP and the designed graph relation transformation network are used to extract unimodal spatiotemporal information of bimodal features. Then, the similarity scores between different unimodal features are calculated and used as bimodal association features. Specifically, the graph relation transformation network of a specific modality can learn long-range dependencies within the modality in dynamic spatial features. At the same time, the present invention designs a graph relation transformation network to learn the temporal relationship interactions within the spatial features of three different frames of the unimodality. Next, the present invention calculates the similarity scores between visual and audio or text features as bimodal association features. Secondly, the present invention designs a multimodal attention feature fusion module to determine the contribution of each feature and aggregate these features. Finally, the present invention trains the MLP model and uses it to predict different personality trait scores. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0100] Figure 1It is a flow chart of a multimodal personality prediction method based on pre-training and graph relation transformation network provided by an embodiment of the present invention;

[0101] Figure 2 is a structural diagram of a video-based explicit personality analysis method provided by an embodiment of the present invention;

[0102] Figure 3 It is a block diagram of a multimodal personality prediction device based on pre-training and graph relation transformation network provided by an embodiment of the present invention;

[0103] Figure 4 It is a structural schematic diagram of a multimodal personality prediction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0104] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0105] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0106] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0107] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0108] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0109] The embodiment of the present invention provides a multimodal personality prediction method based on pre-training and graph relation transformation network, which can be implemented by a multimodal personality prediction device, which can be a terminal or a server. Figure 1 The flowchart of the multimodal personality prediction method based on pre-training and graph relation transformation network is shown. The processing flow of the method may include the following steps:

[0110] S1. Get input video.

[0111] S2. Input the input video to a preprocessing module to obtain visual modality information data, text modality information data, and audio modality information data.

[0112] In a feasible implementation manner, the overall structure of the method of the present invention is as follows Figure 2 As shown in the figure, the input short video is divided into three modal information data after preprocessing: visual, audio and text.

[0113] S3. Input the visual modal information data, the textual modal information data and the audio modal information data into the feature extraction module respectively to obtain visual sequence features, audio sequence features and textual sequence features.

[0114] In a feasible implementation, the present invention uses frozen CLIP and Wav2CLIP encoders to extract multimodal spatial features.

[0115] Optionally, the feature extraction module in S3 includes a visual encoder, an audio encoder, and a text encoder.

[0116] The above step S3 may include the following steps S31-S33:

[0117] S31. Input the visual modality information data into the visual encoder to obtain visual sequence features; wherein the visual encoder is a CLIP visual encoder.

[0118] Optionally, the above step S31 may include the following steps S311-S314:

[0119] S311, dividing the visual modal information data into equal intervals to obtain divided visual modal information data.

[0120] S312. For each frame of scene image in the divided visual modality information data, a spatial scene feature sequence is obtained through the CLIP visual encoder.

[0121] S313. For the spatial scene feature sequence, extract sequence information through xLSTM to obtain spatiotemporal scene features.

[0122] S314. For the spatiotemporal scene features, visual sequence features are obtained through global average pooling and a 1-layer MLP network.

[0123] In a feasible implementation, first, the visual data is divided into 15 frames of scene images at equal intervals. Then, through the CLIP visual encoder, a spatial scene feature sequence is obtained for each frame of the scene image. Next, the present invention uses the high-performance xLSTM (Extended Long Short-Term Memory) to extract the sequence information of these spatial scene features and obtain the spatiotemporal scene features. Finally, the scene features are obtained by global average pooling and a layer of MLP (Multilayer Perceptron) network aggregation and compression features. .

[0124] S32. Input the audio modal information data into an audio encoder to obtain audio sequence features; wherein the audio encoder is a Wav2CLIP audio encoder.

[0125] In a feasible implementation, the present invention first uses the ffmpeg tool to convert the video into audio data to extract audio features. After a series of operations such as framing, pre-emphasis, windowing, fast Fourier transform, Mel filtering and logarithmic transformation, the audio data is converted into 15 equally spaced logarithmic Mel spectrograms with a shape of [60 × 64], denoted as Then, all log-mel spectrograms are fed into the audio encoder of Wav2CLIP to generate spatial audio sequence features. .

[0126] S33. Input the text modal information data into a text encoder to obtain text sequence features; wherein the text encoder is a CLIP text encoder.

[0127] In one feasible implementation, the present invention first preprocesses the input video to obtain the transcribed text. The text is divided into several sentences; the maximum number of sentences is 13, and the features of sentences with less than 13 are filled with zero vectors. Next, these sentences are input into the text encoder of CLIP to obtain text sequence features .

[0128] S4. Input the visual sequence features, audio sequence features and text sequence features into the graph relation transformation network respectively to obtain visual features, audio features and text features, obtain visual-audio association features based on the visual features and audio features, and obtain visual-text association features based on the visual features and text features.

[0129] In a feasible implementation, the present invention proposes a method for multimodal personality prediction. The method extracts two bimodal correlation features based on a designed graph relational transformation network to effectively capture the relational interactions in single-modal features by modeling longer temporal dependencies within the modality.

[0130] Optionally, the step of inputting the visual sequence features, the audio sequence features, and the text sequence features into the graph relation transformation network to obtain the visual features, the audio features, and the text features in S4 may include the following steps S41-S45:

[0131] S41. Obtain position codes for the visual sequence features, audio sequence features and text sequence features respectively according to sine and cosine, and embed the position codes into the corresponding visual sequence features, audio sequence features and text sequence features to obtain visual sequence features after position code embedding, audio sequence features after position code embedding and text sequence features after position code embedding.

[0132] In a feasible implementation, the present invention designs a graph relation transformation network for learning longer temporal interaction dynamics within a modality. The network consists of two modules; the first module is a 1-layer GCN (Graph Convolution Neural Networks), and the second module is a 1-layer transformation encoder.

[0133] S42. Each time series in the visual sequence features after position coding embedding, the audio sequence features after position coding embedding, and the text sequence features after position coding embedding is regarded as a spatial node, and the cosine similarity function is used to calculate the spatial feature similarity between the spatial nodes.

[0134] In one possible implementation, for any input feature ,in , represents a time series, represents the feature dimension of each frame. The present invention first obtains the position code according to the sine and cosine, and adds them to embed them into Then, the present invention regards each time series as a spatial node. Next, the present invention uses the cosine similarity function to calculate the spatial feature similarity between these nodes.

[0135] S43. According to the spatial feature similarity between spatial nodes, an adjacency matrix is ​​constructed using the k-NN algorithm.

[0136] In a feasible implementation, the present invention uses the k-NN algorithm to construct the adjacency matrix A, where and .

[0137] S44. The adjacency matrix is ​​transmitted and feature information is aggregated through a one-layer graph convolutional network to obtain visual output features, audio output features, and text output features.

[0138] The present invention defines a 1-layer GCN to transmit and aggregate feature information. The calculation formula of GCN is as follows:

[0139] (1)

[0140] In the formula, Represents the output features after passing through a 1-layer graph convolutional network and batch normalization operation, represents a batch normalization layer, Degree matrix The diagonal matrix of Represents the adjacency matrix with self-loop I added , represents the input features, represents the weight term of the graph convolution of modality s, represents the bias term of the graph convolution of modality s, represents the output features of mode s, represents the ReLU activation function, represents a time series, Represents the feature dimension of each frame.

[0141] S45. Obtain visual features, audio features, and text features according to the visual output features, audio output features, text output features, and a 1-layer transform encoder.

[0142] In a feasible implementation manner, a layer 1 transform encoder specifically includes:

[0143] (2)

[0144] in is the activation function ReLU, LN is the layer normalization (LayerNorm) layer, and They are the weights and biases of the two fully connected layers. The multi-head attention module MultiHead is as follows:

[0145] (3)

[0146] in and Represent multi-head attention and attention layer respectively. , and represents the learnable parameter matrix. The queries, keys, and values ​​are Q, K, and V, respectively. In addition, the number of heads is , Represents the dimension of the query.

[0147] After passing through a layer of MLP After the dimension is reduced to 128, the present invention uses global average pooling to obtain spatial-temporal features :

[0148] (4)

[0149] In the formula, is the activation function ReLU, and are the weights and bias terms of the fully connected layer, Represents global average pooling.

[0150] Optionally, obtaining visual-audio correlation features according to the visual features and the audio features in S4 includes:

[0151] Through the improved cosine similarity function, the similarity scores of visual features and audio features are calculated bit by bit, and the visual-audio association features are obtained according to the similarity scores.

[0152] In a feasible implementation, based on the scene sequence features obtained previously and audio sequence features , which are input into the graph relation transformation network to obtain scene features and audio features Next, the present invention adopts scene features and audio features The improved cosine similarity function of is used to calculate their similarity scores bit by bit, and these similarity scores are used as scene-audio association features. The improved cosine similarity function is defined as follows:

[0153] (5)

[0154] In the formula, represents the index of each feature, Represents the similarity score of visual features and audio features, Represents visual features, Represents audio features, is a minimum value that is used to prevent division by zero.

[0155] Similarly, the scene features obtained by the graph relation transformation network and text features , and use the improved cosine similarity function to calculate their similarity scores, which are used as visual-text association features Similarly, the visual-textual association feature is calculated as follows:

[0156] (6)

[0157] S5. Input the visual features, visual-audio association features, and visual-text association features into the multimodal attention feature fusion module to obtain fused features.

[0158] In a feasible implementation, the present invention proposes a multimodal attention feature fusion network to obtain complementary information and learn the contributions of different features. The multimodal attention feature fusion network of the present invention obtains the contributions of different features and multiplies these contributions with these features to obtain the final fusion feature. Contribution value It can be calculated as follows:

[0159] (7)

[0160] in , , are the original representations of visual-audio association features, visual scene features, and visual-text association features, respectively. Here, [.] represents a stacking operation. is average pooling, and Represent the weights and bias terms of the two linear layers respectively. In addition, is the softmax activation function. Then, the attention score α and the feature F are weighted and summed to obtain the output of the i-th head For multiple heads, the final feature is the concatenation of the features of each head. This process can be formalized as:

[0161] (8)

[0162] In the formula, Indicates The output of the head, represents the weighted sum operation, represents the fusion feature, Represents a concatenation operation.

[0163] S6. Input the fused features into the prediction head module to obtain the multimodal personality prediction results.

[0164] In a feasible implementation, the present invention uses a two-layer perceptron (MLP) as a personality prediction model. For this prediction model, the first linear layer is activated using the ReLU function, and a dropout layer is added after the activation. The last layer uses the sigmoid function to map the model's prediction output to the range of [0,1].

[0165] Optionally, the method further includes a multimodal loss module. The present invention uses a multi-task loss function to jointly optimize the entire method, which can be defined as follows:

[0166] (9)

[0167] In the formula, represents the multi-task loss function of the multimodal loss module, represents the root mean square error loss function, represents the Log Cosh loss function, represents the Bell loss function, Represents the Gaussian loss function.

[0168] The multimodal personality prediction method based on the pre-trained model and the graph relation transformation network disclosed in the present invention can be used to predict people's Big Five personality trait scores. The method can be widely used in the fields of crime detection, medical rehabilitation, talent recruitment, etc.

[0169] In an embodiment of the present invention, multimodal personality analysis aims to identify people's personality traits in short videos. Based on the fact that the visual modality plays a dominant role in this task, the present invention proposes a dynamic personality prediction method adapted to learning and fusing intra-modal and inter-modal features. The method of the present invention combines a unimodal visual scene feature and two bimodal association features in a multimodal learning method, which can model spatial and temporal feature relationships. First, the present invention uses an encoder of a pre-trained model to extract unimodal spatial scene features from a video. Then, xLSTM is used to capture long-range sequence dependencies between different scene frames, which are used as scene features. The encoders of CLIP and Wav2CLIP and the designed graph relation transformation network are used to extract unimodal spatiotemporal information of bimodal features. Then, the similarity scores between different unimodal features are calculated and used as bimodal association features. Specifically, the graph relation transformation network of a specific modality can learn long-range dependencies within the modality in dynamic spatial features. At the same time, the present invention designs a graph relation transformation network to learn the temporal relationship interaction within the spatial features of three different frames of the unimodality. Next, the present invention calculates the similarity scores between visual and audio or text features as bimodal association features. Secondly, the present invention designs a multimodal attention feature fusion module to determine the contribution of each feature and aggregate these features. Finally, the present invention trains the MLP model and uses it to predict different personality trait scores.

[0170] Figure 3 The block diagram of a device for predicting a personality based on pre-training and a graph relation transformation network multimodal personality according to an exemplary embodiment is shown. The device is used for predicting a personality based on pre-training and a graph relation transformation network multimodal personality. Figure 3The device includes an acquisition module 310, a preprocessing module 320, a feature extraction module 330, a graph relationship transformation module 340, a multimodal attention feature fusion module 350 and a prediction head module 360. Among them:

[0171] The acquisition module 310 is used to acquire input video.

[0172] The preprocessing module 320 is used to input the input video into the preprocessing module to obtain visual modality information data, text modality information data and audio modality information data.

[0173] The feature extraction module 330 is used to input the visual modality information data, the text modality information data and the audio modality information data into the feature extraction module respectively to obtain visual sequence features, audio sequence features and text sequence features.

[0174] The graph relation transformation module 340 is used to input the visual sequence features, audio sequence features and text sequence features into the graph relation transformation network respectively, obtain visual features, audio features and text features, obtain visual-audio association features based on the visual features and audio features, and obtain visual-text association features based on the visual features and text features.

[0175] The multimodal attention feature fusion module 350 is used to input the visual features, the visual-audio association features and the visual-text association features into the multimodal attention feature fusion module to obtain fused features.

[0176] The prediction head module 360 ​​is used to input the fusion features into the prediction head module to obtain a multimodal personality prediction result.

[0177] In an embodiment of the present invention, multimodal personality analysis aims to identify people's personality traits in short videos. Based on the fact that the visual modality plays a dominant role in this task, the present invention proposes a dynamic personality prediction method adapted to learning and fusing intra-modal and inter-modal features. The method of the present invention combines a unimodal visual scene feature and two bimodal association features in a multimodal learning method, which can model spatial and temporal feature relationships. First, the present invention uses an encoder of a pre-trained model to extract unimodal spatial scene features from a video. Then, xLSTM is used to capture long-range sequence dependencies between different scene frames, which are used as scene features. The encoders of CLIP and Wav2CLIP and the designed graph relation transformation network are used to extract unimodal spatiotemporal information of bimodal features. Then, the similarity scores between different unimodal features are calculated and used as bimodal association features. Specifically, the graph relation transformation network of a specific modality can learn long-range dependencies within the modality in dynamic spatial features. At the same time, the present invention designs a graph relation transformation network to learn the temporal relationship interaction within the spatial features of three different frames of the unimodality. Next, the present invention calculates the similarity scores between visual and audio or text features as bimodal association features. Secondly, the present invention designs a multimodal attention feature fusion module to determine the contribution of each feature and aggregate these features. Finally, the present invention trains the MLP model and uses it to predict different personality trait scores.

[0178] Figure 4 is a schematic diagram of the structure of a multimodal personality prediction device provided by an embodiment of the present invention, such as Figure 4 As shown, the multimodal personality prediction device may include the above Figure 3 The multimodal personality prediction device based on pre-training and graph relation transformation network is shown. Optionally, the multimodal personality prediction device 410 may include a first processor 2001.

[0179] Optionally, the multimodal personality prediction device 410 may further include a memory 2002 and a transceiver 2003 .

[0180] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0181] Combine the following Figure 4 The components of the multimodal personality prediction device 410 are specifically introduced as follows:

[0182] The first processor 2001 is the control center of the multimodal personality prediction device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0183] Optionally, the first processor 2001 may perform various functions of the multimodal personality prediction device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0184] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0185] In a specific implementation, as an embodiment, the multimodal personality prediction device 410 may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0186] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0187] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0188] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0189] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0190] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0191] It should be noted that Figure 4 The structure of the multimodal personality prediction device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0192] In addition, the technical effects of the multimodal personality prediction device 410 can refer to the technical effects of the multimodal personality prediction method based on pre-training and graph relation transformation network described in the above method embodiment, which will not be repeated here.

[0193] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0194] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0195] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0196] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0197] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0198] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0199] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0200] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0201] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0202] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0203] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0204] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0205] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal personality prediction method based on pre-training and graph relation transformation network, characterized in that: The method comprises: S1, obtain input video; S2, inputting the input video into a preprocessing module to obtain visual modality information data, text modality information data, and audio modality information data; S3, inputting the visual modality information data, the text modality information data and the audio modality information data into a feature extraction module respectively to obtain visual sequence features, audio sequence features and text sequence features; S4, inputting the visual sequence features, audio sequence features and text sequence features into a graph relation transformation network respectively to obtain visual features, audio features and text features, obtaining visual-audio association features according to the visual features and audio features, and obtaining visual-text association features according to the visual features and text features; S5, inputting the visual features, the visual-audio associated features, and the visual-text associated features into a multimodal attention feature fusion module to obtain fused features; S6. Input the fusion features into the prediction head module to obtain a multimodal personality prediction result.

2. The multimodal personality prediction method based on pre-training and graph relation transformation network according to claim 1 is characterized in that: The feature extraction module in S3 includes a visual encoder, an audio encoder and a text encoder; The step of inputting the visual modality information data, the text modality information data and the audio modality information data into a feature extraction module to obtain visual sequence features, audio sequence features and text sequence features in S3 includes: S31, inputting the visual modality information data into the visual encoder to obtain visual sequence features; wherein the visual encoder is a CLIP visual encoder; S32, inputting the audio modal information data into the audio encoder to obtain audio sequence features; wherein the audio encoder is a Wav2CLIP audio encoder; S33. Input the text modal information data into the text encoder to obtain text sequence features; wherein the text encoder is a CLIP text encoder.

3. The multimodal personality prediction method based on pre-training and graph relation transformation network according to claim 2 is characterized in that: The step of inputting the visual modality information data into the visual encoder to obtain visual sequence features in S31 includes: S311, dividing the visual modality information data into equal intervals to obtain divided visual modality information data; S312, for each frame of scene image in the divided visual modality information data, obtain a spatial scene feature sequence through a CLIP visual encoder; S313, extracting sequence information from the spatial scene feature sequence through xLSTM to obtain spatiotemporal scene features; S314 , obtaining visual sequence features through global average pooling and a 1-layer MLP network for the spatiotemporal scene features.

4. The multimodal personality prediction method based on pre-training and graph relation transformation network according to claim 1, characterized in that: The step of inputting the visual sequence features, the audio sequence features and the text sequence features into the graph relation transformation network to obtain the visual features, the audio features and the text features in S4 includes: S41, respectively obtaining position codes for the visual sequence features, audio sequence features and text sequence features according to sine and cosine, and embedding the position codes into the corresponding visual sequence features, audio sequence features and text sequence features to obtain visual sequence features after position code embedding, audio sequence features after position code embedding and text sequence features after position code embedding; S42, treating each time series in the visual sequence features after position coding embedding, the audio sequence features after position coding embedding, and the text sequence features after position coding embedding as a spatial node, and using the cosine similarity function to calculate the spatial feature similarity between the spatial nodes; S43, constructing an adjacency matrix using a k-NN algorithm according to the spatial feature similarity between the spatial nodes; S44, transmitting and aggregating feature information of the adjacency matrix through a one-layer graph convolutional network to obtain visual output features, audio output features, and text output features; The calculation formula of the graph convolutional network is shown in the following formula (1): （1） In the formula, Represents the output features after passing through a 1-layer graph convolutional network and batch normalization operation, represents a batch normalization layer, Degree matrix The diagonal matrix of Represents the adjacency matrix with self-loop I added , represents the input features, represents the weight term of the graph convolution of modality s, represents the bias term of the graph convolution of modality s, represents the output features of mode s, represents the ReLU activation function, represents a time series, Represents the feature dimension of each frame; S45, obtaining visual features, audio features and text features according to the visual output features, audio output features, text output features and a layer of transform encoder; The calculation formula of the transform encoder is as shown in the following formula (2): （2） In the formula, Represents space-time characteristics, represents the global average pooling layer, represents the weight term of the fully connected layer, Represents the bias term of the fully connected layer.

5. The multimodal personality prediction method based on pre-training and graph relation transformation network according to claim 1, characterized in that: The step of obtaining visual-audio correlation features according to the visual features and the audio features in S4 includes: By using an improved cosine similarity function, similarity scores of the visual features and the audio features are calculated bit by bit, and visual-audio correlation features are obtained according to the similarity scores; The improved cosine similarity function is shown in the following formula (3): （3） In the formula, Represents the similarity score of visual features and audio features, Represents visual features, Represents audio features, ; The step of obtaining visual-text association features according to the visual features and the text features in S4 includes: Through the improved cosine similarity function, the similarity scores of visual features and text features are calculated bit by bit, and the visual-text association features are obtained according to the similarity scores. .

6. The multimodal personality prediction method based on pre-training and graph relation transformation network according to claim 1, characterized in that: The step of inputting the visual features, the visual-audio associated features and the visual-text associated features into a multimodal attention feature fusion module to obtain fused features includes: S51, respectively obtain the contribution value of the visual feature, the contribution value of the visual-audio associated feature, and the contribution value of the visual-text associated feature through the multimodal attention feature fusion module, as shown in the following formula (4): （4） In the formula, Indicates characteristics, Indicates stacking operation, represents the visual-audio association feature, Represents visual features, represents the visual-text association feature, represents the global average pooling output, represents global average pooling, represents the contribution value, represents the softmax activation function, represents the weight term of the first linear layer, represents the bias term of the first linear layer, represents the weight term of the second linear layer, represents the bias term of the second linear layer; S52, according to the contribution value and the visual feature, the visual-audio association feature and the visual-text association feature, a fusion feature is obtained, as shown in the following formula (5): （5） In the formula, Indicates The output of the head, represents the weighted sum operation, represents the fusion feature, Represents a concatenation operation.

7. The multimodal personality prediction method based on pre-training and graph relation transformation network according to claim 1, characterized in that: The method further includes a multimodal loss module, as shown in the following equation (6): （6） In the formula, represents the multi-task loss function of the multimodal loss module, represents the root mean square error loss function, represents the Log Cosh loss function, represents the Bell loss function, Represents the Gaussian loss function.

8. A multimodal personality prediction device based on pre-training and graph relation transformation network, the multimodal personality prediction device based on pre-training and graph relation transformation network is used to implement the multimodal personality prediction method based on pre-training and graph relation transformation network as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, used to acquire input video; A preprocessing module, used for inputting the input video into the preprocessing module to obtain visual modality information data, text modality information data and audio modality information data; A feature extraction module, used to input the visual modality information data, the text modality information data and the audio modality information data into the feature extraction module respectively to obtain visual sequence features, audio sequence features and text sequence features; A graph relation transformation module, used for inputting the visual sequence features, audio sequence features and text sequence features into a graph relation transformation network respectively, obtaining visual features, audio features and text features, obtaining visual-audio association features according to the visual features and audio features, and obtaining visual-text association features according to the visual features and text features; A multimodal attention feature fusion module, used for inputting the visual features, the visual-audio association features and the visual-text association features into the multimodal attention feature fusion module to obtain fused features; The prediction head module is used to input the fusion features into the prediction head module to obtain a multimodal personality prediction result.

9. A multimodal personality prediction device, characterized in that: The multimodal personality prediction device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images