Multimodal personality prediction method based on pre-training model and pyramid graph fusion

Abstract

The present invention provides a multimodal personality prediction method and device based on pre-training model and pyramid graph fusion, and relates to the field of emotional computing technology. The method comprises: according to the original video and the preset personality description word list, a single-modal feature extraction is performed through a pre-training model to obtain three types of single-modal features; based on the similarity function, the similarity between two single-modal features is calculated according to the three types of single-modal features to obtain bimodal association features; according to the text data and the three types of single-modal features, feature splicing is performed through a feature encoder to obtain a trimodal combination feature; according to the three types of single-modal features, bimodal association features and trimodal combination features, feature fusion is performed through a pyramid graph fusion network to obtain multimodal fusion features; input into a double-layer perceptron to perform personality prediction, and obtain a personality prediction result. The present invention is a multimodal personality prediction method with high accuracy that integrates a pre-training model and a pyramid graph fusion network.

Description

Technical Field

[0001] The present invention relates to the field of emotional computing technology, and in particular to a multimodal personality prediction method and device based on the fusion of a pre-training model and a pyramid graph. Background Art

[0002] Personality Personality combines all attributes, including behavior, temperament, emotion, and mental aspects, which together define a person's unique character. In the field of personality computing, the most widely used is the so-called five-factor model. This model includes five traits, which can be abbreviated by acronyms (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism, OCEAN), namely openness, conscientiousness, extraversion, agreeableness, and emotionality. These traits are mainly related to cognitive, emotional, and non-verbal behaviors, and are considered to be relatively stable, affecting an individual's thinking, emotions, and behavior, shaping interpersonal and environmental interactions. There are many applications for personality prediction systems based on artificial intelligence. In human resources, personality traits can be used to assess the match between candidates and positions; in psychological medicine, clinical psychologists focus on studying the role of personality traits in general mental disorders.

[0003] From the perspective of actual performance, the existing excellent algorithms mainly consider the singleness of personality prediction datasets, use pre-trained models to fully extract single modality features, and fuse features of different modalities through different feature fusion strategies. Although these pre-trained models have been trained on medium-sized datasets, their ability to extract features still needs to be improved due to the limited number of samples in the pre-training datasets. At the same time, although the personality prediction problem is a multimodal analysis task, the existing excellent algorithms ignore the exploration of the associated features of different modalities, thus failing to further improve the accuracy of personality prediction.

[0004] In the prior art, there is a lack of a multimodal personality prediction method with high accuracy that integrates a pre-trained model and a pyramid graph fusion network. Summary of the invention

[0005] In order to solve the technical problems of the limited number of samples and neglect of the correlation features between different modalities in the prior art, the embodiment of the present invention provides a multimodal personality prediction method and device based on the fusion of pre-trained model and pyramid graph. The technical solution is as follows:

[0006] On the one hand, a multimodal personality prediction method based on the fusion of a pre-trained model and a pyramid graph is provided, the method is implemented by a multimodal personality prediction device, and the method includes:

[0007] Use a video acquisition device to record a video to obtain an original video; perform data preprocessing on the original video to obtain scene data, audio data, and text data;

[0008] Based on the extended long short-term memory network, according to the scene data, the audio data and a preset personality description word list, a pre-trained model is used to extract unimodal features to obtain three types of unimodal features; the pre-trained model includes a pre-trained CLIP model and a pre-trained CLAP model;

[0009] Based on the similarity function, the similarity between the two unimodal features is calculated according to the three types of unimodal features to obtain three bimodal association features;

[0010] Based on the pre-trained model, the universal feature encoder and the two-layer perceptron, feature fusion is performed according to the text data and the three types of single-modal features to obtain a tri-modal combined feature;

[0011] According to the three types of single-modal features, the bimodal association features and the trimodal combination features, feature fusion is performed through a pyramid graph fusion network to obtain multimodal fusion features;

[0012] Among them, the step of performing feature fusion through a pyramid graph fusion network according to the three types of single-modal features, the bimodal association features and the trimodal combination features to obtain multimodal fusion features includes:

[0013] Constructing a static multimodal graph according to the three types of single-modal features, the bimodal association features and the trimodal combination features;

[0014] According to the static multimodal graph, feature fusion is performed through a pyramid graph fusion network to obtain multimodal fusion features;

[0015] The multimodal fusion features are input into a double-layer perceptron to perform personality prediction and obtain a personality prediction result.

[0016] On the other hand, a multimodal personality prediction device based on the fusion of a pre-trained model and a pyramid graph is provided, and the device is applied to a multimodal personality prediction method based on the fusion of a pre-trained model and a pyramid graph, and the device comprises:

[0017] The data acquisition module is used to record the video using a video acquisition device to obtain the original video; perform data preprocessing on the original video to obtain scene data, audio data and text data;

[0018] A unimodal feature extraction module, for extracting unimodal features based on an extended long short-term memory network, according to the scene data, the audio data and a preset personality description word list, through a pre-trained model to obtain three types of unimodal features; the pre-trained model includes a pre-trained CLIP model and a pre-trained CLAP model;

[0019] A bimodal correlation feature calculation module, used for calculating the similarity between two single modalities based on the three types of single modal features based on a similarity function, to obtain three bimodal correlation features;

[0020] A trimodal combination feature acquisition module, which is used to perform feature fusion according to the text data and the three types of single-modal features based on a pre-trained model, a universal feature encoder and a two-layer perceptron to obtain a trimodal combination feature;

[0021] A multimodal feature fusion module, used to perform feature fusion through a pyramid graph fusion network according to the three types of single-modal features, the bimodal association features and the trimodal combination features, so as to obtain multimodal fusion features;

[0022] Wherein, the multimodal feature fusion module is further used for:

[0023] Constructing a static multimodal graph according to the three types of single-modal features, the bimodal association features and the trimodal combination features;

[0024] According to the static multimodal graph, feature fusion is performed through a pyramid graph fusion network to obtain multimodal fusion features;

[0025] The personality prediction module is used to input the multimodal fusion features into a double-layer perceptron to perform personality prediction and obtain a personality prediction result.

[0026] On the other hand, a multimodal personality prediction device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the multimodal personality prediction methods based on the fusion of a pre-trained model and a pyramid graph is implemented.

[0027] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned multimodal personality prediction methods based on the fusion of a pre-trained model and a pyramid graph.

[0028] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0029] The present invention proposes a multimodal personality prediction method based on the fusion of a pre-trained model and a pyramid graph. By combining multimodal fusion features, a pre-trained model and a pyramid graph fusion network, it is used for explicit personality analysis, further explores multimodal correlation features, and improves the accuracy of personality prediction. Multimodal information covers visual, audio and textual modalities. In order to better obtain information within the modality and enhance cross-modal correlation connections between modalities, the present invention mines single-modal, bimodal and trimodal features based on two large-scale multimodal pre-trained models CLIP and CLAP, and uses the inherent associations between modalities to obtain rich multimodal features. A static multimodal graph and a corresponding pyramid graph fusion network are used to fuse multimodal features, effectively capturing contextual information within the modality and complementary information between modalities. According to the fused multimodal features, an MLP model is used to predict personality traits. The present invention is a multimodal personality prediction method with high accuracy that integrates a pre-trained model and a pyramid graph fusion network. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 It is a flow chart of a multimodal personality prediction method based on the fusion of a pre-training model and a pyramid graph provided by an embodiment of the present invention;

[0032] Figure 2 It is a schematic diagram of the structure of a pyramid graph fusion network provided by an embodiment of the present invention;

[0033] Figure 3 It is a block diagram of a multimodal personality prediction device based on the fusion of a pre-training model and a pyramid graph provided by an embodiment of the present invention;

[0034] Figure 4 It is a structural schematic diagram of a multimodal personality prediction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0036] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0037] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0038] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0039] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0040] The embodiment of the present invention provides a multimodal personality prediction method based on the fusion of a pre-trained model and a pyramid graph. The method can be implemented by a multimodal personality prediction device, which can be a terminal or a server. Figure 1 The flowchart of the multimodal personality prediction method based on the fusion of the pre-trained model and the pyramid graph is shown. The processing flow of the method may include the following steps:

[0041] S1. Use a video acquisition device to record a video and obtain an original video; perform data preprocessing on the original video to obtain scene data, audio data, and text data.

[0042] In a feasible implementation, the recorded original video can be preprocessed using Bili2text, segmented audio can be extracted from the original video, and the speech can be converted into text using OpenAI's Whisper model. The text is divided into several sentences with a maximum number of 13 sentences, and the features of less than 13 sentences are filled with zero vectors.

[0043] S2. Based on the extended long short-term memory network, according to the scene data, audio data and a preset personality descriptor word list, unimodal feature extraction is performed through a pre-trained model to obtain three types of unimodal features; the pre-trained model includes a pre-trained CLIP model and a pre-trained CLAP model.

[0044] Optionally, based on the extended long short-term memory network, according to the scene data, audio data and a preset personality description word list, a pre-trained model is used to perform unimodal feature extraction to obtain three types of unimodal features, including:

[0045] Based on the extended long short-term memory network, the scene features are obtained by extracting features through the image encoder of the pre-trained CLIP model according to the scene data;

[0046] According to a preset personality description word list, feature extraction is performed through a first text encoder of a pre-trained CLIP model to obtain a first text feature;

[0047] Based on the ffmpeg tool and the extended long short-term memory network, feature extraction is performed based on the audio data through the audio encoder of the pre-trained CLAP model to obtain audio features;

[0048] According to a preset personality description word list, feature extraction is performed through a second text encoder of a pre-trained CLAP model to obtain a second text feature;

[0049] The first text feature and the second text feature are concatenated to obtain personality descriptor features.

[0050] In a feasible implementation manner, the contrastive language-image pre-training model (CLIP) adopted by the present invention shows excellent performance on various text-video downstream tasks.

[0051] CLIP consists of two sub-networks, an image encoder and a first text encoder. The image encoder is built on top of the Vision Transformer (VIT), which applies self-attention to image patches and uses the openai / clip-vit-large-patch14-336 model based on the CLIP and ViT architectures. The first text encoder is based on the Bidirectional Encoder Representations from Transformers (BERT), a Transformer framework with architectural modifications that applies self-attention to word tags. The two encoders of CLIP process visual data separately. and text data Input, ( , ) is one of the visual-text pairs arranged by index i, and the processing process is as follows (1) and (2):

[0052] (1);

[0053] (2);

[0054] in, Represents the image encoding of CLIP; Represents the image encoding of CLIP.

[0055] Contrastive Language-Audio Pre-training (CLAP) can develop audio representations through contrastive learning by combining audio data with natural language descriptions. CLAP is pre-trained on the large audio-text dataset LAION-Audio-630K.

[0056] CLAP's two encoders process audio data separately and text data Input, ( , ) is one of the audio-text pairs with index i. Audio Embedding and text embedding Audio encoder with projection layer and the second text encoder The process is as follows:

[0057] (3);

[0058] (4);

[0059] The audio / text projection layer is a two-layer multilayer perceptron that uses ReLU as the activation function to map the encoder output to the same dimension D (i.e., , ).

[0060] Extract 15 frames of scene images at equal intervals from the input original video , use CLIP's image encoder to obtain scene features of scene images. For each frame of scene image, its scene features are based on Scene embedding , the dimension is [1,768], and i is the index of the i-th frame scene image.

[0061] The extended long short-term memory network (xLSTM) is used to extract temporal features and aggregate scene features in an average manner along the time direction to obtain ; A multilayer perceptron (MLP) is used to reduce the dimension of the scene feature vector to .

[0062] xLSTM includes 1 matrix long short-term memory network (Matrix Long Short-Term Memory ScalarLong Short-Term Memory, mLSTM) and 6 scalar long short-term memory networks (Scalar Long Short-TermMemor, sLSTM), and adopts a 4-head attention mechanism.

[0063] The audio data is converted into a logarithmic Mel spectrogram using the ffmpeg tool after a series of operations such as framing, pre-emphasis, windowing, fast Fourier transform, Mel filtering, and logarithmic transform. 15 equally spaced logarithmic Mel spectrograms are extracted for the audio data, with a shape of [60 × 64], denoted as For each log-mel spectrogram, it is fed into the audio encoder of CLAP to generate its audio embedding , the dimension is [1, 512], and j is the index of the logarithmic Mel-spectrogram of the jth frame.

[0064] Embed the obtained audio into Input to the xLSTM model (the same xLSTM used in scene features) to extract temporal features; a layer of MLP is used to reduce audio features Dimension.

[0065] The personality descriptor list used in this invention contains personality descriptors selected from personality surveys and multiple dictionaries, ensuring the diversity and representativeness of the descriptions to effectively capture the complexity of human personality. The personality descriptor list includes 1,608 adjectives that can accurately describe human personality. For each personality descriptor, its embedding is generated using the text encoder of CLIP. As the first text feature, calculate the embedding of all description words The average value of , and use a layer of MLP to reduce its dimension, and get .

[0066] Generate another embedding using CLAP’s text encoder As the second text feature, calculate the embedding of all description words , and use a layer of MLP to reduce its dimension, and get . The above two features ( and ) and use a layer of MLP to reduce the dimension of the combined personality descriptor features to obtain text features .

[0067] S3. Based on the similarity function, the similarity between the two unimodal features is calculated according to the three types of unimodal features to obtain three bimodal association features.

[0068] Optionally, based on the similarity function, similarity between two unimodal features is calculated according to three types of unimodal features to obtain three bimodal association features, including:

[0069] According to the scene feature and the first text feature, a similarity function is used to calculate to obtain a scene-personality descriptor similarity score; the scene-personality descriptor similarity score is used as a scene-personality descriptor association feature;

[0070] According to the audio feature and the second text feature, a similarity function is used to calculate to obtain an audio-personality descriptor similarity score; the audio-personality descriptor similarity score is used as an audio-personality descriptor association feature;

[0071] Based on the scene features and audio features, a similarity function is used to calculate and obtain a scene-audio similarity score; the scene-audio similarity score is used as a scene-audio association feature.

[0072] In one feasible implementation, based on the personality descriptor features obtained through the CLIP text encoder and scene features obtained through the CLIP image encoder , by introducing the similarity function to calculate the personality description features bit by bit and scene features and use these similarity scores as scene-personality descriptor association features The similarity function is defined as follows (5):

[0073] (5);

[0074] Among them, i represents the index of each feature vector, Represents one as 10 -8 to prevent division by 0.

[0075] Similar to the obtained scene-personality descriptor association feature, the difference between the audio-personality descriptor association feature is that the encoder used for the audio-personality descriptor association feature is the encoder of CLAP. and personality description features based on CLAP text encoder and, using the similarity function to calculate their similarity scores, which are used as audio-personality descriptor association features .

[0076] Since both audio and visual modalities are non-verbal, aligning these two modalities in time series is more effective for extracting scene-audio correlation features. Previously, the present invention has extracted 15 frames of aligned scene images and 15 frames of audio mel spectrograms according to time series. Therefore, the present invention obtains scene features through the visual encoder of CLIP , and the audio features are obtained through the CLAP audio encoder , by computing the scene embedding and audio embed The similarity score is used as the scene-audio association feature. .

[0077] S4. Based on the pre-trained model, universal feature encoder and two-layer perceptron, feature fusion is performed according to text data and three types of single-modal features to obtain tri-modal combined features.

[0078] Optionally, based on the pre-trained model, the universal feature encoder and the two-layer perceptron, feature fusion is performed according to the text data and the three types of single-modal features to obtain a tri-modal combined feature, including:

[0079] According to the text data, feature extraction is performed through the first text encoder of the CLIP model to obtain a third text feature;

[0080] Performing dimensionality reduction processing on the scene features, the third text features, and the audio features to obtain processed scene features, processed text features, and processed audio features with the same dimensions;

[0081] Input the processed scene features, processed text features and processed audio features into a universal feature encoder for feature extraction to obtain scene universal features, text universal features and audio universal features; the universal feature encoder is a six-layer stacked Transformer encoder;

[0082] The scene universal features, text universal features and audio universal features are concatenated to obtain trimodal concatenated features;

[0083] The three-modal concatenation features are input into the double-layer perceptron to obtain the three-modal combination features.

[0084] In a feasible implementation, based on the scene embedding obtained in the above steps and audio embed . Input the original text segmented sentences into the first text encoder of CLIP to obtain text embedding As the third text feature. The feature embeddings of the three modalities are reduced to the same dimension through a layer of MLP to obtain .

[0085] In order to obtain the trimodal combined feature (VAT), the present invention uses a standard six-layer transformer encoder as a universal feature encoder. The three inputs are embedded into the universal encoder to obtain their respective universal features, denoted as , the process is as follows (6):

[0086] (6);

[0087] in, , are learnable parameters.

[0088] Based on the calculated general features, the refined features of each modality are obtained in a fixed dimension through an average operation and a layer of MLP, which is denoted as The three common features are concatenated and sent to the two-layer MLP to obtain the three-modal combined features .

[0089] In order to make the features of the three common features similar, the present invention uses the contrast loss function as the constraint condition for each pair of features as follows (7):

[0090] (7);

[0091] Where N is the total number of positive samples, defined as two common features belonging to the same video, and negative samples are defined as two common features not belonging to the same video.

[0092] The present invention randomly constructs N negative samples, so there are 2N sets of feature pairs. (s1, s2) represents any pair of combinations in {v, a, t}, is the label of the i-th group of samples. If it is a positive sample, it is 1, if it is a negative sample, it is 0. represents the Euclidean distance of the i-th group of feature pairs. Therefore, the final contrast loss function is calculated as follows (8):

[0093] (8);

[0094] S5. Based on the three types of single-modal features, bimodal association features and trimodal combination features, feature fusion is performed through a pyramid graph fusion network to obtain multimodal fusion features.

[0095] Among them, according to three types of single-modal features, bimodal association features and trimodal combination features, feature fusion is performed through the pyramid graph fusion network to obtain multimodal fusion features, including:

[0096] Construct a static multimodal graph based on three types of unimodal features, bimodal association features, and trimodal combination features;

[0097] According to the static multimodal graph, feature fusion is performed through the pyramid graph fusion network to obtain multimodal fusion features.

[0098] In a feasible implementation, the present invention constructs a static multimodal graph represented as G = (U, E), where U represents a node set, each element (i.e., node) represents a specific modal feature, and E represents an edge set between nodes. For example, Through Multi-Head Graph Attention Networks (MultiGATs), information of different modalities (i.e. Visual information and The calculation process of MultiGAT is as follows (9) and (10):

[0099] (9);

[0100] (10);

[0101] in, , L is the number of network layers; MultiGAT stands for multi-head graph attention layer; It is The feature matrix of the layer, E is the edge set, which are all inputs of MultiGATs; || represents the concatenation operation, is a trainable parameter, and H is the output of the pyramid graph fusion network (Mixed Potential Green Function, MPGFN) of the present invention.

[0102] Among them, the nodes of the static multimodal graph include scene features, personality descriptor features, audio features, scene-personality descriptor association features, audio-personality descriptor association features, scene-audio association features and trimodal combination features;

[0103] The edges of a static multimodal graph include homomodal edges and cross-modal edges.

[0104] In a feasible implementation manner, the present invention uses unimodal features including scene features (V), audio features (A), and text features (T); bimodal features including scene-personality description association features (VT), audio-personality description association features (AT), and scene-audio association features (VA); and trimodal combination features (VAT) as nodes of MPGFN.

[0105] There are two types of edges in static multimodal graphs, namely homomodal edges and cross-modal edges. Each node is connected to the relevant nodes of the current modal feature, that is, homomodal edges. Each node is connected to nodes belonging to different modal features but from the same modal type, that is, cross-modal edges. Figure 2 In the node Not only connect related nodes (same modal edge), also connecting nodes and (cross-modal edges).

[0106] Among them, the pyramid graph fusion network includes 4 multi-head graph attention networks and 1 cognitive pyramid network;

[0107] The multi-head graph attention network is used to capture the interactive information between homomodal features and cross-modal features.

[0108] In a feasible implementation, since the graph attention network has a strong ability in modeling relationships, the present invention uses MultiGATs and cognitive pyramid structures to establish the relationship between multimodal features and fuse them into a comprehensive feature, namely, a pyramid graph fusion network.

[0109] like Figure 2 As shown in Figure 2, it consists of four MultiGATs for extracting intra-modal and inter-modal interaction information. The present invention connects the output of the previous layer of the network to the input of the next layer. The present invention also retains the outputs of all layers and then concatenates them to obtain the final result. The input of each MultiGAT includes information from features of the same modality type and related modalities.

[0110] S6. Input the multimodal fusion features into the double-layer perceptron for personality prediction to obtain the personality prediction result.

[0111] In a feasible implementation, the present invention uses a two-layer perceptron to predict the five personality traits. For this prediction model, the ReLU activation function is used to activate the first linear layer, and a dropout layer is added after the activation. The last layer uses a sigmoid activation function to map the model's predicted output to the range of [0,1].

[0112] The present invention uses a multi-task loss function to jointly optimize the entire framework. The loss function is defined as follows (11):

[0113] (11);

[0114] in, is the root mean square error loss (RMSE); is the log hyperbolic cosine loss; It is Bell loss; is the Gaussian loss. Among them, β is a hyperparameter set to 0.01.

[0115] The present invention proposes a multimodal personality prediction method based on the fusion of a pre-trained model and a pyramid graph. By combining multimodal fusion features, a pre-trained model and a pyramid graph fusion network, it is used for explicit personality analysis, further explores multimodal correlation features, and improves the accuracy of personality prediction. Multimodal information covers visual, audio and textual modalities. In order to better obtain information within the modality and enhance cross-modal correlation connections between modalities, the present invention mines single-modal, bimodal and trimodal combination features based on two large-scale multimodal pre-trained models CLIP and CLAP, and uses the inherent associations between modalities to obtain rich multimodal features. A static multimodal graph and a corresponding pyramid graph fusion network are used to fuse multimodal features, effectively capturing contextual information within the modality and complementary information between modalities. According to the fused multimodal features, an MLP model is used to predict personality traits. The present invention is a multimodal personality prediction method with high accuracy that integrates a pre-trained model and a pyramid graph fusion network.

[0116] Figure 3 The present invention is a block diagram of a multimodal personality prediction device based on the fusion of a pre-trained model and a pyramid graph according to an exemplary embodiment. The device is used in a multimodal personality prediction method based on the fusion of a pre-trained model and a pyramid graph. Figure 3 The device includes a data acquisition module 310, a single-modal feature extraction module 320, a dual-modal associated feature calculation module 330, a tri-modal combined feature acquisition module 340, a multi-modal feature fusion module 350 and a personality prediction module 360. Among them:

[0117] The data acquisition module 310 is used to record a video using a video acquisition device to obtain an original video; perform data preprocessing on the original video to obtain scene data, audio data, and text data;

[0118] The unimodal feature extraction module 320 is used to extract unimodal features based on the extended long short-term memory network, according to the scene data, the audio data and the preset personality description word list, through the pre-trained model to obtain three types of unimodal features; the pre-trained model includes a pre-trained CLIP model and a pre-trained CLAP model;

[0119] A bimodal correlation feature calculation module 330 is used to calculate the similarity between two unimodal features based on a similarity function and three types of unimodal features to obtain three bimodal correlation features;

[0120] A trimodal combination feature acquisition module 340 is used to perform feature fusion according to text data and three types of single-modal features based on a pre-trained model, a universal feature encoder and a two-layer perceptron to obtain a trimodal combination feature;

[0121] The multimodal feature fusion module 350 is used to perform feature fusion through a pyramid graph fusion network according to three types of single-modal features, bimodal association features and trimodal combination features to obtain multimodal fusion features;

[0122] The multimodal feature fusion module 350 is further used for:

[0123] Construct a static multimodal graph based on three types of unimodal features, bimodal association features, and trimodal combination features;

[0124] According to the static multimodal graph, feature fusion is performed through the pyramid graph fusion network to obtain multimodal fusion features;

[0125] The personality prediction module 360 ​​is used to input the multimodal fusion features into the double-layer perceptron to perform personality prediction and obtain a personality prediction result.

[0126] Optionally, the single-modal feature extraction module 320 is further used to:

[0127] Based on the extended long short-term memory network, the scene features are obtained by extracting features through the image encoder of the pre-trained CLIP model according to the scene data;

[0128] According to a preset personality description word list, feature extraction is performed through a first text encoder of a pre-trained CLIP model to obtain a first text feature;

[0129] Based on the ffmpeg tool and the extended long short-term memory network, feature extraction is performed based on the audio data through the audio encoder of the pre-trained CLAP model to obtain audio features;

[0130] According to a preset personality description word list, feature extraction is performed through a second text encoder of a pre-trained CLAP model to obtain a second text feature;

[0131] The first text feature and the second text feature are concatenated to obtain personality descriptor features.

[0132] Optionally, the bimodal association feature calculation module 330 is further used to:

[0133] According to the scene feature and the first text feature, a similarity function is used to calculate to obtain a scene-personality descriptor similarity score; the scene-personality descriptor similarity score is used as a scene-personality descriptor association feature;

[0134] According to the audio feature and the second text feature, a similarity function is used to calculate to obtain an audio-personality descriptor similarity score; the audio-personality descriptor similarity score is used as an audio-personality descriptor association feature;

[0135] Based on the scene features and audio features, a similarity function is used to calculate and obtain a scene-audio similarity score; the scene-audio similarity score is used as a scene-audio association feature.

[0136] Optionally, the three-modal combined feature acquisition module 340 is further used to:

[0137] According to the text data, feature extraction is performed through the first text encoder of the CLIP model to obtain a third text feature;

[0138] Performing dimensionality reduction processing on the scene features, the third text features, and the audio features to obtain processed scene features, processed text features, and processed audio features with the same dimensions;

[0139] Input the processed scene features, processed text features and processed audio features into a universal feature encoder for feature extraction to obtain scene universal features, text universal features and audio universal features; the universal feature encoder is a six-layer stacked Transformer encoder;

[0140] The scene universal features, text universal features and audio universal features are concatenated to obtain trimodal concatenated features;

[0141] The three-modal concatenation features are input into the double-layer perceptron to obtain the three-modal combination features.

[0142] Among them, the nodes of the static multimodal graph include scene features, personality descriptor features, audio features, scene-personality descriptor association features, audio-personality descriptor association features, scene-audio association features and trimodal combination features;

[0143] The edges of a static multimodal graph include homomodal edges and cross-modal edges.

[0144] Among them, the pyramid graph fusion network includes 4 multi-head graph attention networks and 1 cognitive pyramid network;

[0145] The multi-head graph attention network is used to capture the interactive information between homomodal features and cross-modal features.

[0146] The present invention proposes a multimodal personality prediction method based on the fusion of a pre-trained model and a pyramid graph. By combining multimodal fusion features, a pre-trained model and a pyramid graph fusion network, it is used for explicit personality analysis, further explores multimodal correlation features, and improves the accuracy of personality prediction. Multimodal information covers visual, audio and textual modalities. In order to better obtain information within the modality and enhance cross-modal correlation connections between modalities, the present invention mines single-modal, bimodal and trimodal features based on two large-scale multimodal pre-trained models CLIP and CLAP, and uses the inherent associations between modalities to obtain rich multimodal features. A static multimodal graph and a corresponding pyramid graph fusion network are used to fuse multimodal features, effectively capturing contextual information within the modality and complementary information between modalities. According to the fused multimodal features, an MLP model is used to predict personality traits. The present invention is a multimodal personality prediction method with high accuracy that integrates a pre-trained model and a pyramid graph fusion network.

[0147] Figure 4 is a schematic diagram of the structure of a multimodal personality prediction device provided by an embodiment of the present invention, such as Figure 4 As shown, the multimodal personality prediction device may include the above Figure 3 The multimodal personality prediction device based on the fusion of pre-trained model and pyramid graph is shown. Optionally, the multimodal personality prediction device 410 may include a first processor 2001.

[0148] Optionally, the multimodal personality prediction device 410 may further include a memory 2002 and a transceiver 2003 .

[0149] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0150] Combine the following Figure 4 The components of the multimodal personality prediction device 410 are specifically introduced as follows:

[0151] The first processor 2001 is the control center of the multimodal personality prediction device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0152] Optionally, the first processor 2001 may perform various functions of the multimodal personality prediction device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0153] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0154] In a specific implementation, as an embodiment, the multimodal personality prediction device 410 may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0155] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0156] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0157] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0158] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0159] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0160] It should be noted that Figure 4 The structure of the multimodal personality prediction device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0161] In addition, the technical effects of the multimodal personality prediction device 410 can refer to the technical effects of the multimodal personality prediction method based on the fusion of the pre-trained model and the pyramid graph described in the above method embodiment, which will not be repeated here.

[0162] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0163] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0164] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0165] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0166] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0167] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0168] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0173] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0174] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal personality prediction method based on the fusion of pre-trained model and pyramid graph, characterized in that: The method comprises: Use a video acquisition device to record a video to obtain an original video; perform data preprocessing on the original video to obtain scene data, audio data, and text data; Based on the extended long short-term memory network, according to the scene data, the audio data and a preset personality description word list, a pre-trained model is used to extract unimodal features to obtain three types of unimodal features; the pre-trained model includes a pre-trained CLIP model and a pre-trained CLAP model; Based on the similarity function, the similarity between the two unimodal features is calculated according to the three types of unimodal features to obtain three bimodal association features; Based on the pre-trained model, the universal feature encoder and the two-layer perceptron, feature fusion is performed according to the text data and the three types of single-modal features to obtain a tri-modal combined feature; According to the three types of single-modal features, the bimodal association features and the trimodal combination features, feature fusion is performed through a pyramid graph fusion network to obtain multimodal fusion features; Among them, the step of performing feature fusion through a pyramid graph fusion network according to the three types of single-modal features, the bimodal association features and the trimodal combination features to obtain multimodal fusion features includes: Constructing a static multimodal graph according to the three types of single-modal features, the bimodal association features and the trimodal combination features; According to the static multimodal graph, feature fusion is performed through a pyramid graph fusion network to obtain multimodal fusion features; The multimodal fusion features are input into a double-layer perceptron to perform personality prediction and obtain a personality prediction result.

2. The multimodal personality prediction method based on the fusion of pre-training model and pyramid graph according to claim 1, characterized in that: Based on the extended long short-term memory network, according to the scene data, the audio data and the preset personality description word list, the unimodal feature extraction is performed through the pre-trained model to obtain three types of unimodal features, including: Based on the extended long short-term memory network, according to the scene data, feature extraction is performed through the image encoder of the pre-trained CLIP model to obtain scene features; According to a preset personality description word list, feature extraction is performed through a first text encoder of a pre-trained CLIP model to obtain a first text feature; Based on the ffmpeg tool and the extended long short-term memory network, according to the audio data, feature extraction is performed through the audio encoder of the pre-trained CLAP model to obtain audio features; According to a preset personality description word list, feature extraction is performed through a second text encoder of a pre-trained CLAP model to obtain a second text feature; The first text feature and the second text feature are concatenated to obtain a personality descriptor feature.

3. The multimodal personality prediction method based on the fusion of pre-training model and pyramid graph according to claim 2, characterized in that: Based on the similarity function, the similarity between the two single modalities is calculated according to the three types of single modal features to obtain three bimodal association features, including: Using a similarity function to calculate based on the scene feature and the first text feature, a scene-personality descriptor similarity score is obtained; and the scene-personality descriptor similarity score is used as a scene-personality descriptor association feature; Using a similarity function to calculate based on the audio feature and the second text feature, an audio-personality descriptor similarity score is obtained; and the audio-personality descriptor similarity score is used as an audio-personality descriptor association feature; A similarity function is used to perform calculation according to the scene feature and the audio feature to obtain a scene-audio similarity score; and the scene-audio similarity score is used as a scene-audio association feature.

4. The multimodal personality prediction method based on the fusion of pre-training model and pyramid graph according to claim 2, characterized in that: The method of performing feature fusion based on the pre-trained model, the universal feature encoder and the two-layer perceptron according to the text data and the three types of single-modal features to obtain a tri-modal combined feature includes: According to the text data, feature extraction is performed by a first text encoder of a CLIP model to obtain a third text feature; Performing dimensionality reduction processing on the scene feature, the third text feature, and the audio feature to obtain processed scene features, processed text features, and processed audio features with the same dimensions; Inputting the processed scene features, processed text features and processed audio features into a universal feature encoder for feature extraction to obtain scene universal features, text universal features and audio universal features; the universal feature encoder is a six-layer stacked Transformer encoder; Performing feature splicing on the scene universal feature, the text universal feature and the audio universal feature to obtain a trimodal splicing feature; The three-modal concatenation features are input into a double-layer perceptron to obtain the three-modal combination features.

5. The multimodal personality prediction method based on the fusion of pre-training model and pyramid graph according to claim 1, characterized in that: The nodes of the static multimodal graph include scene features, personality descriptor features, audio features, scene-personality descriptor association features, audio-personality descriptor association features, scene-audio association features, and trimodal combination features; The edges of the static multimodal graph include same-modal edges and cross-modal edges.

6. The multimodal personality prediction method based on the fusion of pre-training model and pyramid graph according to claim 1, characterized in that: The pyramid graph fusion network includes 4 multi-head graph attention networks and 1 cognitive pyramid network; The multi-head graph attention network is used to capture the interactive information between homomodal features and cross-modal features.

7. A multimodal personality prediction device based on the fusion of a pre-trained model and a pyramid graph, wherein the multimodal personality prediction device based on the fusion of a pre-trained model and a pyramid graph is used to implement the multimodal personality prediction method based on the fusion of a pre-trained model and a pyramid graph as claimed in any one of claims 1 to 6, characterized in that: The device comprises: The data acquisition module is used to record the video using a video acquisition device to obtain the original video; perform data preprocessing on the original video to obtain scene data, audio data and text data; A unimodal feature extraction module, for extracting unimodal features based on an extended long short-term memory network, according to the scene data, the audio data and a preset personality description word list, through a pre-trained model to obtain three types of unimodal features; the pre-trained model includes a pre-trained CLIP model and a pre-trained CLAP model; A bimodal correlation feature calculation module, used for calculating the similarity between two single modalities based on the three types of single modal features based on a similarity function, to obtain three bimodal correlation features; A trimodal combination feature acquisition module, which is used to perform feature fusion according to the text data and the three types of single-modal features based on a pre-trained model, a universal feature encoder and a two-layer perceptron to obtain a trimodal combination feature; A multimodal feature fusion module, used to perform feature fusion through a pyramid graph fusion network according to the three types of single-modal features, the bimodal association features and the trimodal combination features, so as to obtain multimodal fusion features; Wherein, the multimodal feature fusion module is further used for: Constructing a static multimodal graph according to the three types of single-modal features, the bimodal association features and the trimodal combination features; According to the static multimodal graph, feature fusion is performed through a pyramid graph fusion network to obtain multimodal fusion features; The personality prediction module is used to input the multimodal fusion features into a double-layer perceptron to perform personality prediction and obtain a personality prediction result.

8. The multimodal personality prediction device based on the fusion of pre-training model and pyramid graph according to claim 7, characterized in that: The three-modal combined feature acquisition module is further used for: According to the text data, feature extraction is performed through the first text encoder of the CLIP model to obtain a third text feature; Performing dimensionality reduction processing on the scene features, the third text features, and the audio features to obtain processed scene features, processed text features, and processed audio features with the same dimensions; Input the processed scene features, processed text features and processed audio features into a universal feature encoder for feature extraction to obtain scene universal features, text universal features and audio universal features; the universal feature encoder is a six-layer stacked Transformer encoder; The scene universal features, text universal features and audio universal features are concatenated to obtain trimodal concatenated features; The three-modal concatenation features are input into the double-layer perceptron to obtain the three-modal combination features.

9. A multimodal personality prediction device, characterized in that: The multimodal personality prediction device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images