A multimodal personality prediction method and system integrating spatiotemporal graph attention network

By integrating the space-time graph attention network and the cross-modal attention fusion module, the problems of improper facial feature mining and unreasonable modal weight allocation are solved, and the detailed prediction of personality traits is achieved, which is suitable for multiple application fields.

CN118228194BActive Publication Date: 2024-11-08UNIV OF SCI & TECH BEIJING +1
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202410393379.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-02
Publication Date
2024-11-08
Estimated Expiration
2044-04-02

AI Technical Summary

Technical Problem

The prior art does not mine face features in personality detection, and the weight allocation is unreasonable during multimodal fusion, making it difficult to accurately evaluate the importance of features between modals.

Method used

The fusion space-time graph attention network is used to extract the scene space-time appearance characteristics, face space-time appearance characteristics, geometric structure characteristics, audio characteristics and text characteristics in the video through the convolutional neural network, and adaptive allocation is used to utilize the cross-modal attention fusion module. Weight, highlight important characteristics and suppress weak characteristics.

Benefits of technology

It realizes the meticulous exploration of face features and the reasonable weight allocation of multimodal features, improves the accuracy and reliability of personality prediction, and can intelligently and quantitatively predict the Big Five personality traits.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118228194B_ABST
    Figure CN118228194B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal personality prediction method and system integrating spatiotemporal graph attention network, which relates to the field of emotional computing technology, including: extracting spatiotemporal appearance features of scenes, spatiotemporal appearance features of faces and audio features in target videos based on convolutional neural network models; extracting spatiotemporal geometric structural features of faces in target videos based on graph attention network models; extracting text features of target videos based on pre-trained language models; fusing spatiotemporal appearance features of scenes, spatiotemporal appearance features of faces, spatiotemporal geometric structural features of faces, audio features and text features to obtain multimodal fusion features; predicting the personality of subjects based on multimodal fusion features and trained personality prediction models to obtain the personality trait scores of subjects. The present invention alleviates the technical problems existing in the prior art that the features of faces are not mined meticulously and the weight distribution between modalities is unreasonable.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of emotional computing technology, and in particular to a multimodal personality prediction method and system integrating a spatiotemporal graph attention network. Background Art

[0002] Personality is a combination of all the attributes of behavior, temperament, emotion, and psychology, and is a characteristic of a unique individual. Generally speaking, personality is used to describe a person's psychological state. In the past few decades, some researchers have established various personality trait models using psychological scales, such as the Big Five personality model, 16PF, MBTI, etc. The Big Five personality model aims to measure a person's personality through five traits, namely openness, conscientiousness, extroversion, agreeableness, and neuroticism. In the process of communication, people's potential personality traits are usually expressed through facial expressions, body movements, and conversation styles. Automatically predicting a person's personality from a conversation video and providing timely and objective help for downstream practical applications has gradually become a demand. Although there are many multimodal methods containing video, audio, and text for personality detection, most of the existing work focuses on learning the appearance knowledge of the visual modality based on the CNN architecture, and the feature mining of the face is not detailed, and the dynamics of the geometric knowledge based on the face image has not been fully utilized. In addition, the existing methods for multimodal fusion are mostly simple serial combinations, and the importance of different modalities to the final result is not considered, resulting in unreasonable weight distribution between modalities. Summary of the invention

[0003] In order to solve the above technical problems existing in the prior art, the embodiment of the present invention provides a multimodal personality prediction method and system integrating a spatiotemporal graph attention network. The technical solution is as follows:

[0004] On the one hand, a multimodal personality prediction method integrating a spatiotemporal graph attention network is provided, the method comprising: extracting spatiotemporal appearance features of scenes in a target video based on a first convolutional neural network model; the target video is a video including a conversation of a subject; extracting spatiotemporal appearance features of faces in the target video based on a second convolutional neural network model; extracting spatiotemporal geometric structural features of faces in the target video based on a graph attention network model; extracting audio features of the target video based on a third convolutional neural network model; extracting text features of the target video based on a pre-trained language model; performing feature fusion on the spatiotemporal appearance features of the scenes, the spatiotemporal appearance features of faces, the spatiotemporal geometric structural features of faces, the audio features and the text features to obtain multimodal fusion features; performing personality prediction on the subject based on the multimodal fusion features and a trained personality prediction model to obtain a personality trait score of the subject.

[0005] Optionally, the first convolutional neural network model is a convolutional neural network model that combines a ResNet-18 network and a BiGRU network.

[0006] Optionally, the second convolutional neural network model is a convolutional neural network model that combines a VGGFace model and a BiGRU network; wherein the VGGFace model serves as a feature extractor for each frame image in the target video, and uses a DAN+ method to extract features; and the BiGRU network is used to extract timing information.

[0007] Optionally, the graph attention network model includes multiple spatiotemporal graph convolution modules and attention modules; the spatiotemporal graph convolution module includes a spatial module and a temporal module, the spatial module includes a spatial graph convolution layer, a first batch of sample normalization layer and a first activation layer, and the temporal module includes a convolution layer, a second batch of sample normalization layer and a second activation layer.

[0008] Optionally, the third convolutional neural network model includes a VGGish model; extracting the audio features of the target video based on the third convolutional neural network model includes: acquiring an audio signal of the target video; preprocessing the audio signal to obtain a processed audio signal; and extracting a potential feature vector of the processed audio signal based on a pre-trained VGGish model to obtain the audio features.

[0009] Optionally, the audio signal is preprocessed to obtain a processed audio signal, including: resampling the audio signal to obtain a mono signal; calculating a spectrogram of the mono signal based on a Hamming window; mapping the spectrogram to a Mel spectrogram; calculating a logarithmic Mel spectrogram based on the Mel spectrogram; and constructing non-overlapping samples based on the logarithmic Mel spectrogram to obtain the processed audio signal.

[0010] Optionally, the personality prediction model includes a multi-layer perceptron.

[0011] On the other hand, a multimodal personality prediction system integrating a spatiotemporal graph attention network is also provided, comprising: a first extraction module, a second extraction module, a third extraction module, a fourth extraction module, a fifth extraction module, a fusion module and a prediction module;

[0012] Among them, the first extraction module is used to extract the scene spatiotemporal appearance features in the target video based on the first convolutional neural network model; the target video is a video including the subject's conversation; the second extraction module is used to extract the face spatiotemporal appearance features in the target video based on the second convolutional neural network model; the third extraction module is used to extract the face spatiotemporal geometric structure features in the target video based on the graph attention network model; the fourth extraction module is used to extract the audio features of the target video based on the third convolutional neural network model; the fifth extraction module is used to extract the text features of the target video based on the pre-trained language model; the fusion module is used to fuse the scene spatiotemporal appearance features, the face spatiotemporal appearance features, the face spatiotemporal geometric structure features, the audio features and the text features to obtain multimodal fusion features; the prediction module is used to predict the personality of the subject based on the multimodal fusion features and the trained personality prediction model to obtain the personality trait score of the subject.

[0013] On the other hand, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0014] On the other hand, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the above method is implemented.

[0015] The embodiment of the present invention provides a multimodal personality prediction method and system integrating a spatiotemporal graph attention network, which utilizes a spatiotemporal facial graph and a spatiotemporal graph attention network to integrate the geometric features and appearance features of video frame images, and uses the facial graph topology to capture dynamic spatiotemporal features for personality prediction. The present invention designs a cross-modal attention fusion module, which can adaptively assign weights to different modal representations, highlight stronger representations and suppress weaker representations according to their importance, thereby alleviating the technical problems of the prior art of not mining facial features in detail and irrational weight distribution between modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1It is a flowchart of a multimodal personality prediction method integrating a spatiotemporal graph attention network provided by an embodiment of the present invention;

[0018] Figure 2 Schematic diagram of the architecture of the first convolutional neural network model provided by an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of the architecture of a second convolutional neural network model provided in an embodiment of the present invention;

[0020] Figure 4 It is a schematic diagram of the architecture of the graph attention network model provided by an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of the structure of a feature extraction unit provided by an embodiment of the present invention;

[0022] Figure 6 Schematic diagram of the architecture of a pre-trained language model provided by an embodiment of the present invention;

[0023] Figure 7 It is a schematic diagram of a multimodal personality prediction system integrating a spatiotemporal graph attention network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0025] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0026] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0027] Embodiment 1

[0028] Figure 1 is a flowchart of a multimodal personality prediction method integrating a spatiotemporal graph attention network according to an embodiment of the present invention. Figure 1 As shown, the method specifically comprises the following steps:

[0029] Step S102: extracting spatiotemporal appearance features of scenes in a target video based on a first convolutional neural network model; the target video is a video including a conversation between subjects.

[0030] Step S104: extracting the spatiotemporal appearance features of the face in the target video based on the second convolutional neural network model.

[0031] Step S106: extracting the spatiotemporal geometric structural features of the face in the target video based on the graph attention network model.

[0032] Step S108: extracting audio features of the target video based on the third convolutional neural network model.

[0033] Step S110: extracting text features of the target video based on the pre-trained language model.

[0034] Step S112, fusing scene spatiotemporal appearance features, face spatiotemporal appearance features, face spatiotemporal geometric structure features, audio features and text features to obtain multimodal fusion features.

[0035] Step S114, based on the multimodal fusion features and the trained personality prediction model, the personality of the subject is predicted to obtain the personality trait score of the subject.

[0036] Specifically, in an embodiment of the present invention, the first convolutional neural network model is a convolutional neural network model that combines a ResNet-18 network and a BiGRU network.

[0037] Figure 2 Schematic diagram of the first convolutional neural network model architecture provided according to an embodiment of the present invention. Figure 2 As shown, in order to model the spatiotemporal environmental features in the target video, the present invention adopts a method combining CNN and BiGRU networks. Specifically, the present invention selects the ResNet-18 network as the CNN backbone, and uses it to extract a single-frame scene image feature model. The ResNet-18 network is initialized using weights pre-trained on the places365 scene classification dataset. The present invention deletes the last two layers of the ResNet-18 network, and uses layer3 and layer4 to output features, connecting the fourth feature obtained by the maximum pooling and average pooling operations, which produces a 1536-dimensional scene feature that describes each scene frame. On top of this, two BiGRU layers are placed, including a dropout layer.

[0038] Specifically, in an embodiment of the present invention, the second convolutional neural network model is a convolutional neural network model combining a VGGFace model and a BiGRU network; wherein,

[0039] The VGGFace model is used as the feature extractor for each frame of the target video, and the DAN+ method is used to extract features; the BiGRU network is used to extract timing information.

[0040] Figure 3is a schematic diagram of the architecture of a second convolutional neural network model provided according to an embodiment of the present invention. Figure 3 As shown, the present invention uses VGGFace as the feature extractor for each frame image and uses DAN + The feature extraction method is to select the output of the last and second last convolutional layers of the last convolutional module, and perform average pooling and maximum pooling separately after these two convolutional layers. Therefore, for any frame of face image, a total of 4 feature vectors are obtained, which are then After normalization and concatenation, a high-dimensional feature is finally obtained and used as the feature of the frame. + The features extracted by the method are as follows Figure 3 shown.

[0041] In addition, in order to make the VGGFace model better for personality prediction, the present invention uses the face data extracted from the Big Five personality dataset to fine-tune the model, loads the pre-trained VGGFace model and modifies the output dimension of FC8 to 5, corresponding to the five personality traits of the Big Five, and freezes the convolutional layers of Conv 1, Conv 2, and Conv 3, and only trains the parameters of Conv 4, Conv 5, FC 6, FC 7, and FC 8. After obtaining the video features, the present invention uses BiGRU to extract the timing information.

[0042] Specifically, in an embodiment of the present invention, the graph attention network model includes multiple spatiotemporal graph convolution modules and attention modules; the spatiotemporal graph convolution module includes a spatial module and a temporal module, the spatial module includes a spatial graph convolution layer, a first batch of sample normalization layer and a first activation layer, and the temporal module includes a convolution layer, a second batch of sample normalization layer and a second activation layer.

[0043] Figure 4 Schematic diagram of the architecture of the graph attention network model provided according to an embodiment of the present invention. Figure 4 As shown in the figure, the extraction of face spatiotemporal geometric structure features relies on graph convolutional neural network to extract spatiotemporal graph structure features from key point position features and local visual features.

[0044] For the input key point positions and key point features, two independent graph attention network models are constructed to extract graph structural features about the positions and graph structural features about the local appearance of the key points respectively.

[0045] Specifically, a spatiotemporal graph convolution module is constructed, which includes a spatial module and a temporal module. Among them, a spatial module contains a spatial graph convolution layer, a batch normalization layer, and an activation layer, and a temporal module contains a convolution layer, a batch normalization layer, and an activation layer. The key point position coordinate input passes through two consecutive spatiotemporal graph convolution modules, and the dimension of the position spatiotemporal geometric structure feature generated is [b=1, c=64, t=16, n=20], where b represents the size of a batch of samples participating in training, c represents the feature dimension generated by each node in the graph, t represents the number of frames, and n represents 20 key points obtained in each frame of the face. The key point appearance feature input passes through three consecutive spatiotemporal graph convolution modules, and the dimension of the appearance spatiotemporal geometric structure feature generated is [b=1, c=128, t=16, n=20]. The two obtained spatiotemporal geometric structure features are combined in the channel dimension to obtain the spatiotemporal geometric structure feature [b=1, c=192, t=16, n=20].

[0046] Then create a graph convolution module to further process the spatiotemporal geometric structure features after concatenation. The graph convolution module contains 1 spatial module, 1 channel attention module, 1 temporal module, and 1 inter-frame attention module. The introduction of the channel attention module makes the spatiotemporal feature extraction pay different attention to different features and pay more attention to more important features. The introduction of the inter-frame attention module makes the model pay more attention to more important frames. After being processed by the graph convolution module, the feature dimension obtained is still [b=1, c=192, t=16, n=20]. For this feature, the maximum value and mean operation are used in the spatial dimension, and the two features are combined in the feature dimension to obtain the feature [b=1, c=384, t=16, n=1]. After removing the dimensions other than the b dimension that are 1, the face spatiotemporal graph geometric structure feature is of dimension [b=1, t=16, c=384].

[0047] Specifically, the third convolutional neural network model includes a VGGish model; step S108 also includes the following steps:

[0048] Step S1081, obtaining the audio signal of the target video;

[0049] Step S1082: preprocess the audio signal to obtain a processed audio signal. Specifically, the preprocessing process includes:

[0050] Resample the audio signal to obtain a mono signal;

[0051] Based on the Hamming window, calculate the spectrogram of the mono signal;

[0052] Map the spectrogram to a Mel-spectrogram;

[0053] Based on the Mel-spectrogram, calculate the logarithmic Mel-spectrogram;

[0054] Construct non-overlapping samples based on the logarithmic Mel spectrogram to obtain the processed audio signal.

[0055] Step S1083: Based on the pre-trained VGGish model, extract the potential feature vector of the processed audio signal to obtain the audio feature.

[0056] Specifically, first, preprocess the audio signal. Use soundfile to read the wav format audio file and perform the following processing:

[0057] 1) Resample all audio to 16kHz mono;

[0058] 2) Use a periodic Hamming window of size 25ms and interval 10ms to calculate the spectrogram;

[0059] 3) Then the spectrogram is mapped into a Mel-spectrogram with 64 frequency bins covering the range of 125-7500 Hz;

[0060] 4) Calculate a stable logarithmic Mel-spectrogram to avoid the logarithm being zero, so a certain offset needs to be added;

[0061] 5) These features are constructed as 0.96 second non-overlapping samples, where each sample covers 64 Mel bands and 96 10ms frames.

[0062] After the above preprocessing steps, each 1-second audio can be processed into a two-dimensional feature array with a shape of 96×64. Therefore, for a 15-second video, a total of 15 audio signals are extracted. In order to extract the potential feature vector representation from these audio signals, the present invention uses the pre-trained VGGishCNN as a feature extractor. The VGGish model converts the input audio signal into a 128-dimensional high-level feature vector. After obtaining the audio features from each second of the video, the potential feature vector is obtained. Finally, the audio feature vectors of 15 frames are connected in series as the final feature representation of the audio modality. Among them, the feature extraction unit structure is as follows Figure 5 shown.

[0063] Figure 6 Schematic diagram of the architecture of a pre-trained language model provided according to an embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, the pre-trained language model is a Reberta model. Specifically, Reberta is a variant of the BERT model using a Transformer architecture, including 12 encoder layers and 12 head self-attention mechanisms, which learns language representation through unsupervised learning.

[0064] In the embodiment of the present invention, the text is first processed using the RobertaTokenizer based on the pre-trained Roberta model, and the process is completed in the following steps:

[0065] 1) Preprocessing, such as converting text to lowercase, removing extra spaces, etc.

[0066] 2) Divide the text into tokens based on the constructed word list;

[0067] 3) Add special tags at the beginning and end of the text, such as [CLS], [SEP], etc.

[0068] Then a RoBERTa model is used for construction and fine-tuning, where the RoBERTa model loads the pre-trained model parameters. Finally, the trained RoBERTa model is used to extract a 768-dimensional embedding feature vector representation from the text of the short video.

[0069] Specifically, in an embodiment of the present invention, the personality prediction model includes a multi-layer perceptron.

[0070] Specifically, the five feature vectors obtained are concatenated into one feature vector, which is then used as the input of the module to evaluate the contribution of each feature branch and minimize the information redundancy caused by feature diversity. Here, a residual structure is added to the module to mitigate information loss. The module includes two fully connected layers to calculate the attention weight α of each dimension in the multi-channel representation F. α can be calculated as:

[0071] α=tanh(W 2 tanh(W 1 F+b)+c

[0072] Among them, W 1 , W 2 , b and c represent the weight matrix and bias of the two fully connected layers respectively. Use tanh to limit the attention weight to the interval [-1,1], and then perform element-wise multiplication of the obtained attention vector α with each dimension of the multi-channel vector F. The final fused multimodal feature vector is expressed as follows:

[0073] F=F×α+F

[0074] Among them, F' is the output of the module, which is input into the MLP to predict the five main personality trait scores. The embodiment of the present invention uses a three-layer perceptron as the Big Five personality prediction model. For this prediction model, the activation of the first two linear layers uses the ReLU activation function, and a dropout layer is added after activation. The last layer uses the sigmoid activation function to map the predicted output of the model to the range between [0,1].

[0075] From the above description, it can be seen that the example of the present invention provides a multimodal personality prediction method that integrates the spatiotemporal graph attention network, which can make full use of the dynamics of the geometric knowledge of the face image and more carefully mine the facial features. The cross-modal attention fusion module designed by the present invention can adaptively assign weights to different representations, highlight stronger representations and suppress weaker representations according to their importance. This method can intelligently, quantitatively and conveniently predict people's Big Five personality traits. This method can be widely used in talent selection and career assessment analysis in public security departments, schools, hospitals and other fields.

[0076] Embodiment 2

[0077] Figure 7 Schematic diagram of a multimodal personality prediction system integrating a spatiotemporal graph attention network according to an embodiment of the present invention. Figure 7 As shown, the system includes: a first extraction module 10, a second extraction module 20, a third extraction module 30, a fourth extraction module 40, a fifth extraction module 50, a fusion module 60 and a prediction module 70.

[0078] Specifically, the first extraction module 10 is used to extract the spatiotemporal appearance features of the scene in the target video based on the first convolutional neural network model; the target video is a video including a conversation between the subjects.

[0079] Specifically, the first convolutional neural network model is a convolutional neural network model that combines a ResNet-18 network and a BiGRU network.

[0080] The second extraction module 20 is used to extract the spatiotemporal appearance features of the face in the target video based on the second convolutional neural network model.

[0081] Specifically, the second convolutional neural network model is a convolutional neural network model that combines the VGGFace model and the BiGRU network; the VGGFace model is used as a feature extractor for each frame image in the target video, and uses the DAN+ method to extract features; the BiGRU network is used to extract timing information.

[0082] The third extraction module 30 is used to extract the spatiotemporal geometric structure features of the face in the target video based on the graph attention network model.

[0083] Specifically, the graph convolutional neural network model includes multiple spatiotemporal graph convolution modules and attention modules; the spatiotemporal graph convolution module includes a spatial module and a temporal module, the spatial module includes a spatial graph convolution layer, a batch normalization layer and an activation layer, and the temporal module includes a convolution layer, a batch normalization layer and an activation layer.

[0084] The fourth extraction module 40 is used to extract audio features of the target video based on the third convolutional neural network model.

[0085] Specifically, the third convolutional neural network model includes a VGGish model; the fourth extraction module 40 is also used to: obtain the audio signal of the target video; preprocess the audio signal to obtain a processed audio signal; based on the pre-trained VGGish model, extract the potential feature vector of the processed audio signal to obtain the audio feature.

[0086] The fifth extraction module 50 is used to extract text features of the target video based on the pre-trained language model.

[0087] Specifically, the pre-trained language model is a Reberta model.

[0088] The fusion module 60 is used to fuse the scene spatiotemporal appearance features, the face spatiotemporal appearance features, the face spatiotemporal geometric structure features, the audio features and the text features to obtain multimodal fusion features.

[0089] The prediction module 70 is used to predict the personality of the subject based on the multimodal fusion features and the trained personality prediction model to obtain the personality trait score of the subject.

[0090] Specifically, the personality prediction model includes a multi-layer perceptron.

[0091] An embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method in the first embodiment when executing the computer program.

[0092] An embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the method in the first embodiment is implemented.

[0093] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0094] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0095] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0097] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this example solution.

[0099] In addition, each functional unit in each example of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0100] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0101] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal personality prediction method integrating spatiotemporal graph attention network, It is characterized in that The method comprises: Based on the first convolutional neural network model, extracting the spatiotemporal appearance features of the scene in the target video; the target video is a video including a conversation of the subject; Extracting the spatiotemporal appearance features of the face in the target video based on the second convolutional neural network model; Extracting the spatiotemporal geometric structural features of the face in the target video based on a graph attention network model; Extracting audio features of the target video based on a third convolutional neural network model; Extracting text features of the target video based on a pre-trained language model; The scene spatiotemporal appearance features, the face spatiotemporal appearance features, the face spatiotemporal geometric structure features, the audio features and the text features are subjected to feature fusion to obtain a multimodal fusion feature; Based on the multimodal fusion features and the trained personality prediction model, predicting the personality of the subject to obtain the personality trait score of the subject; The first convolutional neural network model is a convolutional neural network model combining a ResNet-18 network and a BiGRU network. The ResNet-18 network is selected as the CNN backbone and is used to extract a single-frame scene image feature model. The ResNet-18 network is initialized using weights pre-trained on the places365 scene classification dataset. The last two layers of the ResNet-18 network are deleted, and layer3 and layer4 are used to output features. The fourth feature obtained by the maximum pooling and average pooling operations is connected, which generates a 1536-dimensional scene feature describing each scene frame. On top of this, two BiGRU layers are placed, including a dropout layer. The extraction of spatiotemporal geometric structural features of faces relies on graph convolutional neural networks. The spatiotemporal graph structural features are extracted from key point position features and local visual features. For the input key point positions and key point features, two independent graph attention network models are constructed to extract the graph structural features about the position and the graph structural features about the local appearance of the key points respectively. A spatiotemporal graph convolution module is constructed, which includes a spatial module and a temporal module. A spatial module contains a spatial graph convolution layer, a batch sample normalization layer, an activation layer, and a temporal The module contains 1 convolution layer, 1 batch normalization layer, and 1 activation layer. The key point position coordinate input passes through 2 consecutive spatiotemporal graph convolution modules to generate position spatiotemporal geometric structure feature dimensions of [b=1, c=64, t=16, n=20], where b represents the size of a batch of samples participating in training, c represents the feature dimension generated by each node in the graph, t represents the number of frames, and n represents 20 key points in each frame of the face. The key point appearance feature input passes through 3 consecutive spatiotemporal graph convolution modules to generate appearance spatiotemporal geometric structure features of dimensions of [b=1,c=128,t=16,n=20], the two spatiotemporal geometric structure features are combined in the channel dimension to obtain the spatiotemporal geometric structure features [b=1,c=192,t=16,n=20], and then a graph convolution module is created to further process the spatiotemporal geometric structure features after concatenation. The graph convolution module contains 1 spatial module, 1 channel attention module, 1 temporal module, and 1 inter-frame attention module. The introduction of the channel attention module enables the extraction of spatiotemporal features to pay different attention to different features, paying more attention to more Important features, the introduction of the inter-frame attention module makes the model pay more attention to more important frames. After being processed by the graph convolution module, the feature dimension obtained is still [b=1, c=192, t=16, n=20]. For this feature, the maximum and mean operations are used in the spatial dimension, and the two features are combined in the feature dimension to obtain the feature of [b=1, c=384, t=16, n=1]. After removing the dimensions other than the b dimension that are 1, the geometric structure feature of the face spatiotemporal graph is obtained with a dimension of [b=1, t=16, c=384]. The third convolutional neural network model includes a VGGish model; Use the Roberta Tokenizer based on the pre-trained Roberta model to process the text; The personality prediction model includes a multi-layer perceptron, which concatenates the five obtained feature vectors into one feature vector, which is then used as the input of the module to evaluate the contribution of each feature branch and minimize the information redundancy caused by feature diversity. A residual structure is added to this module to mitigate information loss. It includes two fully connected layers to calculate the attention weight α of each dimension in the multi-channel representation F. α is calculated as: α=tanh(W 2 tanh(W 1 F+b)+c Among them, W 1 , W 2 , b and c represent the weight matrix and bias of the two fully connected layers respectively. Tanh is used to limit the attention weight to the interval [-1,1]. Then the obtained attention vector α is element-wise multiplied with each dimension of the multi-channel vector F. The final fused multimodal feature vector is expressed as follows: F'=F×α+F Among them, F′ is the output of the module, which is input into the MLP to predict the five main personality trait scores. The three-layer perceptron is used as the Big Five personality prediction model. For this prediction model, the activation of the first two linear layers uses the ReLU activation function, and a dropout layer is added after the activation. The last layer uses the sigmoid activation function to map the prediction output of the model to the range between [0,1].

2. The method according to claim 1, It is characterized in that The second convolutional neural network model is a convolutional neural network model combining a VGGFace model and a BiGRU network; wherein, The VGGFace model is used as a feature extractor for each frame of the target video, and uses the DAN+ method to extract features; The BiGRU network is used to extract timing information.

3. The method according to claim 1, It is characterized in that The extracting the audio features of the target video based on the third convolutional neural network model includes: Acquire the audio signal of the target video; Preprocessing the audio signal to obtain a processed audio signal; Based on the pre-trained VGGish model, the potential feature vector of the processed audio signal is extracted to obtain the audio feature.

4. The method according to claim 3, It is characterized in that Preprocessing the audio signal to obtain a processed audio signal includes: Resampling the audio signal to obtain a mono signal; Based on the Hamming window, calculating the spectrogram of the mono signal; Mapping the spectrogram into a Mel-spectrogram; Based on the Mel-spectrogram, calculating a logarithmic Mel-spectrogram; Non-overlapping samples are constructed based on the logarithmic Mel-spectrogram to obtain the processed audio signal.

5. A multimodal personality prediction system integrating a spatiotemporal graph attention network, used to implement the method described in any one of claims 1 to 4, It is characterized in that include: A first extraction module, a second extraction module, a third extraction module, a fourth extraction module, a fifth extraction module, a fusion module and a prediction module; wherein, The first extraction module is used to extract the spatiotemporal appearance features of the scene in the target video based on the first convolutional neural network model; the target video is a video including a conversation of the subject; The second extraction module is used to extract the spatiotemporal appearance features of the face in the target video based on a second convolutional neural network model; The third extraction module is used to extract the spatiotemporal geometric structure features of the face in the target video based on the graph attention network model; The fourth extraction module is used to extract the audio features of the target video based on the third convolutional neural network model; The fifth extraction module is used to extract text features of the target video based on a pre-trained language model; The fusion module is used to fuse the scene spatiotemporal appearance features, the face spatiotemporal appearance features, the face spatiotemporal geometric structure features, the audio features and the text features to obtain multimodal fusion features; The prediction module is used to perform personality prediction on the subject based on the multimodal fusion features and the trained personality prediction model to obtain the personality trait score of the subject.

6. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Micro-expression recognition method based on space-time appearance movement attention network

    CN112307958A

  • User personality feature prediction method and device based on multi-modal information fusion

    CN113705725A

  • Crowd abnormal event detection method based on audio-visual learning

    CN114241408A

  • Multi-modal emotion recognition method based on graph convolutional network

    CN116229225A

  • Text and vision-based time-space correlation type multi-modal emotion recognition method and system

    CN116258989A