A user personality prediction method based on multi-modal data fusion and application

Abstract

The application discloses a user personality prediction method and application based on multi-modal data fusion. The method comprises the following steps: using text data and picture data on a network platform, extracting a text feature vector and a picture feature vector from the text data and the picture data respectively, splicing the text feature vector and the picture feature vector to obtain a spliced vector; using user relationship data on the network platform, taking a user as a node of a graph, constructing edges between nodes according to the user relationship data, taking the spliced vector as a semantic representation of the node, inputting the graph into a user personality prediction model based on a graph neural network, and outputting a user personality prediction value; and dividing text data of the user according to a period, and adjusting the prediction value of the user according to the text data of different periods. The application makes full use of multi-modal data of text, pictures and relationships, and can improve the accuracy of user personality prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information mining, and more particularly to a user personality prediction method based on multi-modal data fusion and application. BACKGROUND

[0002] In recent years, with the rapid development of mobile communication technology, various website platforms characterized by user interaction have been widely popularized. Any behavior of a user on a network platform can reflect the user's preferences and personality characteristics such as character. In particular, social behaviors such as attention, comments, and praise among users can form a user group with similar preferences and personalities. Large network platforms such as video platforms, e-commerce platforms, social platforms, and knowledge platforms are committed to fully mining user interaction rules and building personality portraits of user preferences and personality to enhance the matching degree of user needs, improve user usage, and ultimately improve platform value. At the same time, various website platforms have developed from a text-based mode to a platform that integrates multiple modalities such as images, text, audio, and video. Traditional single-modal information processing methods cannot fully utilize the information of different modalities that implies user relationships and user interaction rules, and it is more necessary to use multi-modal data fusion methods to predict user personality, which is the focus of the present application. Predicting platform user personality has a huge application space in advertising promotion, personalized recommendation, psychological disease treatment, and prevention.

[0003] Network platform user interaction behaviors include self-display behavior, attention behavior, transfer comment and praise behavior, and post behavior. The present application mines the relationship between these behaviors and the user's personality through the image-text information and topological information produced in the self-display behavior, attention behavior, or transfer comment and praise behavior and post behavior.

[0004] Currently, the training data of the personality prediction method applied on the network platform is mostly derived from a single modality, and lacks the feature of attention relationship. Not every user will provide three kinds of information of image, text, and relationship. If the model can only extract one kind of information, it has to clean up the users without content, such as users who only watch others' status without publishing, users who only post text without posting images, etc.

[0005] In addition, when all text information of a single user is one-time feature processing, extreme information will inevitably be averaged into stable information. For example, a user usually only forwards life content without opinions, which will make the user obtain a low score in the N (neuroticism, easy anxiety) dimension, that is, a high score in emotional stability, but when negative social news occurs, the user will make extreme comments, which implies that the user should obtain a high score in the N dimension. However, due to the difference in the number of life content and negative news content, directly counting all text information will average out a small amount of extreme content, finally leading to inaccurate results. SUMMARY

[0006] In view of at least one defect of the prior art or the demand for improvement, the present application provides a user personality prediction method based on multi-modal data fusion, which can improve the accuracy of user personality prediction.

[0007] To achieve the above-mentioned purpose, according to the first aspect of the present application, a user personality prediction method based on multi-modal data fusion is provided, comprising:

[0008] Using the text data and picture data of the user on the network platform, text feature vectors and picture feature vectors are extracted from the text data and picture data respectively, the text feature vectors and picture feature vectors are spliced to obtain the spliced vector of the user;

[0009] Using the user relationship data of the user on the network platform, the user is taken as a node of a graph, the edges between the nodes are constructed according to the relationship data between the users, the spliced vector of the user is taken as the semantic representation of the node, and the constructed graph data is input into the user personality prediction model based on graph neural network to output the user personality prediction value.

[0010] Further, the user personality prediction method based on multi-modal data fusion further comprises: dividing the text data of the user according to period, calculating the emotional instability degree of the user according to the text data of different periods according to a pre-designed calculation formula, and adjusting the prediction value of the user according to the emotional instability degree of the user.

[0011] Further, the emotional instability degree includes an emotional value change degree and a speaking frequency change degree, and the emotional value change degree of the user i is denoted as S i , and the speaking frequency change degree of the user i is denoted as P i ,

[0012] The calculation formula of S i and P i is:

[0013]

[0014]

[0015]

[0016] wherein Y ij+1 is the emotional value of the user i in the j+1th week, Y ij is the emotional value of the user i in the jth week, n is the total number of periods, T ij+1 is the number of posts contained in the text of the user i in the j+1th week, T ij is the number of posts contained in the text of the user i in the jth week.

[0017] Further, the calculation formula for adjusting the predicted value of the user is:

[0018]

[0019] wherein N new represents the predicted value of the user i in neuroticism after adjustment, N ori the predicted value of the user i in neuroticism before adjustment, represents the mean of the degree of change of the speech frequency P of all users, represents the mean of the predicted value N ori of all users in neuroticism, represents the mean of the degree of change of the emotional value S of all users.

[0020] Further, the relationship data of the user on the network platform includes the follow-only nodes and the bidirectional follow nodes of the user, and the user personality prediction model gives different weights to the follow-only nodes and the bidirectional follow nodes of the user respectively when aggregating the relationship data between users.

[0021] Further, the text feature vector is a vector composed of the word frequency of different kinds of words, the picture data is a user avatar, and the picture feature vector is a vector composed of the probability of the user avatar picture belonging to different categories.

[0022] Further, the user personality prediction model includes five prediction sub-models, each of which is used to output the predicted value of the user in a certain dimension of the five personality dimensions in the Big Five Personality Model, and each prediction sub-model is trained separately.

[0023] According to the second aspect of the present application, a user personality prediction system based on multi-modal data fusion is also provided, comprising:

[0024] The feature vector acquisition module is configured to vectorize the text data and the picture data of the user on the network platform, extract a text feature vector and a picture feature vector from the text data and the picture data respectively, splice the text feature vector and the picture feature vector, and obtain a spliced vector of the user.

[0025] The prediction module is configured to use the user relationship data of the user on the network platform, take the user as a node of a graph, construct edges between the nodes according to the relationship data between the users, take the spliced vector of the user as a semantic representation of the node, input the graph into a user personality prediction model based on a graph neural network, and output a user personality prediction value.

[0026] Further, the user personality prediction system based on multi-modal data fusion further comprises:

[0027] The reverse adjustment unit is configured to divide the text data of the user according to a period, calculate an emotional instability degree of the user according to a pre-designed calculation formula based on the text data of different periods, and adjust the prediction value of the user according to the emotional instability degree of the user.

[0028] According to a third aspect of the present application, a storage medium is provided, which stores a computer program executable by a processor, and when the computer program runs on the processor, the processor executes the steps of the method according to any one of the preceding aspects.

[0029] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0030] (1) The semantic information of the node includes the language habits and the preferred image of the user, and the multi-modal information can more comprehensively reflect the personality characteristics, and the prediction accuracy can be improved by using the multi-modal information for user personality prediction.

[0031] (2) The nodes of the graph neural network have the characteristic of information transmission, and the attention preference of the user can also reflect the personality characteristics of the user to a certain extent, and the present application improves the traditional sampling method to obtain the characteristics.

[0032] (3) The prediction value is adjusted by introducing the time feature, which avoids the huge error of the one-time average processing of all the features of a single user without considering the time. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative labor on the basis of these drawings.

[0034] Figure 1 A flowchart of a user personality prediction method based on multi-modal data fusion provided by an embodiment of the present application is shown in the figure.

[0035] Figure 2 A principle diagram of a user personality prediction method based on multi-modal data fusion provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0037] The terms "comprise" and "have" and any variations thereof in the specification and claims of the present application and the above-described accompanying drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but can optionally include steps or modules not listed, or can optionally include other steps or modules inherent to the process, method, product or device.

[0038] As shown in Figure 1 and Figure 2 The user personality prediction method based on multi-modal data fusion of the embodiment of the present application includes steps S101-S102. On the one hand, the user personality prediction can be improved by using multi-modal information. On the other hand, the information transmission between nodes of the graph neural network is used to assist the prediction.

[0039] Further, the user personality prediction method based on multi-modal data fusion of the embodiment of the present application further includes step S103, which adjusts the prediction value by introducing time characteristics, avoiding the huge error of one-time average processing of all text and image features of a single user on the personality without considering time.

[0040] S101, using text data and picture data of a user on a network platform, extracting a text feature vector and a picture feature vector from the text data and the picture data, respectively, and splicing the text feature vector and the picture feature vector to obtain a spliced vector of the user. That is, a preprocessing stage.

[0041] The purpose of the preprocessing stage is to process the media data collected from the Internet by means of crawlers and the like into formatted data and extract features thereof. The main processing objects specifically include text, avatars and the like. The text utilizes the word segmentation technology in natural language processing to obtain the word frequency statistics of the obtained words, and the word frequency will be one of the features of the graph neural network; the avatar picture directly extracts the classification and probability of each class of picture content using the powerful ImageNet library in machine vision, and the probabilities serve as feature vectors, which will also become the node self-feature of the graph neural network.

[0042] Then the obtained text and image feature attributes are converted into a readable format of the graph neural network, that is, cross-modal fusion is performed before the data is input into the neural network. The specific method is to splice the text statistical information and image classification information as a picture semantic vector, and process it into a GNN node self semantic feature.

[0043] The preferred implementation of text feature extraction, image feature extraction and feature splicing will be described below.

[0044] (1) Text feature extraction

[0045] Suppose the data source of the present patent is the crawler results of part of the users of a social media website, and each user can be crawled to obtain user profile, user avatar, user blog, user fans and user attention. Each of these plus a unique identifier id can be abstracted as a user node in the topological graph of the social media relationship network.

[0046] The blog of each user is traversed as a text data set, which is obtained from all the text information on the timeline of each user's homepage and only contains the text of the text.

[0047] If the data is Chinese text, the jieba word segmentation tool needs to be used to first segment the words from the sentence, and if the data is English text, this step can be omitted.

[0048] The present application preferably extracts the word frequency of the feature words in the text by using the LIWC software. Open the LIWC software, input the lemmatized text into the software in units of users, and the software will output the word frequency of each kind of word in the text of each user. There are 15 kinds of word categories, including singular subject (I, myself, me), plural subject (we, us, our), plural third person (they, them), verb (share, do, propose), quantity adjective (some, many, all), modal word (this, that, every), social word (give, call, meet), emotion word (pity, warmth, acuteness), positive emotion (confidence, satisfaction, blessing), negative emotion (worry, suspicion, revenge), uneasy emotion (uneasy, concern, doubt), eating (eat, drink, thirst), achievement (good at, challenge, victory), love (love, kiss, confess), hearing (speak, voice, shout).

[0049] Then, the word frequency of the 15 kinds of categories is combined to form the text feature vector of the user. For example, after counting the word frequency, the number of words in the 15 categories of user A is 10, 15, 17, 3, 8, 11, 28, 12, 19, 5, 0, 0, 5, 13, and 22. Then the text feature vector of user A is (10, 15, 17, 3, 8, 11, 28, 12, 19, 5, 0, 0, 5, 13, 22).

[0050] (2) Image feature extraction

[0051] First, all the pictures are processed to the same resolution, such as 227x227. Then the pictures are input into the neural network, and a pre-trained model can be used to output a classification vector for each picture. Since the picture classification technology has been mature, the present application is a specific implementation of the prior art, and the data source is the crowd annotation data provided by Stanford University, which contains 14 million pictures and more than 20,000 picture categories; the pre-trained model can preferably use ResNet101, and the data source of this model contains 1.2 million pictures and 1000 categories in ImageNet.

[0052] After preparing the pre-trained model, input the user's avatar data into the model and perform classification prediction. The output layer of the model should output the predicted classification, and the present application uses the probability vector of all categories obtained in the full connection layer as the output, without outputting a unique result.

[0053] Since the probabilities of all categories are directly obtained, there is obvious sparsity when the image feature vectors of all users are listed together. Therefore, the image feature vector of each user is taken as a row, n users are n rows, 1000 categories are 1000 columns, and an n*1000 matrix is formed. Then, sparsity filtering is performed, that is, if more than 90% of the element values in a column are below 0.1, the column is deleted. Each row of the filtered matrix is an image feature vector of a user.

[0054] (3) Feature vector splicing

[0055] Suppose the dimension of the filtered image feature vector is m, and the text feature vector of each user is spliced with the image feature vector at the back. The dimension of the complete user feature vector is 15+m.

[0056] In S102, the user is taken as a node of a graph, and edges between nodes are constructed according to the user relationship data, the spliced vector of the user is taken as the semantic representation of the node, and the graph is input into the user personality prediction model based on the graph neural network to output the user personality prediction value.

[0057] The user personality prediction model based on the graph neural network is used to output the personality prediction value of each node / user. Specifically, it can be the prediction value of five personality dimensions in the Big Five Personality Model, including O (experience openness), C (seriousness), E (extroversion), A (agreeableness), and N (neuroticism).

[0058] Further, the user personality prediction model includes five prediction sub-models, each of which is used to output the prediction value of the user in a certain dimension, and each prediction sub-model is trained separately. A neural network only calculates one personality score.

[0059] (1) Edge construction

[0060] Like the input data of most GNN algorithms, in the embodiment of the application, the edge file needs to be prepared as a utf-8 encoded txt or csv file with one node pair per row. The embodiment of the application preferably uses a GraphSage graph neural network. Assume that each user is a node in the topological graph, and each user node has a unique identifier id. A node pair refers to a follow relationship between two nodes. Since GraphSage is a directed graph model, an edge can be constructed as long as there is a one-way follow. In the file, two node ids in the same row are used to represent the relationship, and are separated by an English comma.

[0061] (2) Sampling scheme

[0062] Since it is assumed that the data source of the patent is the crawler result of a social media website, the relationship data can be data describing the follow relationship between users. If the data source is a video platform, an e-commerce platform, or a news platform, the relationship data can be data describing the comment, like relationship, collection relationship, and fan group between users.

[0063] The information of the follow direction is ignored when the edge is constructed, so the follow direction information is added back to the model in the sampling step. Through analysis, it can be concluded that what a user follows and likes determines the personality of a person, and personality is not determined by the followers. Therefore, the information that needs to be passed in the graph neural network only occurs in the behavior of active follow. Therefore, only the follow nodes of the current node are sampled during sampling, including only follow nodes and bidirectional follow nodes, and the followed nodes are ignored. Moreover, when the user personality prediction model aggregates the relationship data between users, different weights are assigned to the only follow nodes and bidirectional follow nodes of the user. Therefore, in the sampling scheme given by the present application, the weight ratio of the bidirectional follow edge, the follow edge, and the followed edge is 6:3:1.

[0064] (3) Aggregation function

[0065] In the GraphSage model, the representation of a node in each layer comes from the aggregation of the representation of the previous layer and the representation of the sampled nodes. On the basis of direct aggregation, the present application adds different weights to the bidirectional follow nodes and the only follow nodes, and the aggregation function is as follows:

[0066]

[0067] Similar to the average aggregation function in GraphSage, the node representation of the next layer is obtained by averaging the representation of the node in the previous layer and the representation of the neighbor nodes in the previous layer. On the basis of this, the present application adds the bidirectional follow friend nodes again to improve their weights. In the formula, v represents the representation of node v in the kth layer; Bi(v) represents the bidirectional follow nodes of v; Si(v) represents the only follow nodes of v; v represents the representation of node v in the kth layer; Bi(v) represents the bidirectional follow nodes of v; Si(v) represents the only follow nodes of v; u represents the representation of node u in the k-1th layer.

[0068] (4) Loss function

[0069] Since the purpose of the present application is not to obtain the representation of the node but to obtain the personality score of the node, the original unsupervised loss function of GraphSage is not selected, and the loss function selected by the embodiment of the present application is the mean square error.

[0070] (5) Training and prediction

[0071] The personality scores of a few people obtained by the questionnaire are used as labels, and the rest of the users are predicted. A graph neural network is trained for each of the five personalities to obtain the score of the personality, i.e. the same user sample needs 5 different score labels, and five training is performed five times to obtain five graph neural network models and prediction results.

[0072] S103, divide the text data of the user by period, calculate the emotional instability degree of the user according to the text data of different periods according to the pre-designed calculation formula, and adjust the prediction value of the user according to the emotional instability degree of the user.

[0073] Specifically, on the basis of the Big Five personality scores obtained by the graph network, the stability of the N dimension score in the time dimension is calculated, and then the stability is compared and adjusted with the score obtained by the graph neural network. The final score is outputted.

[0074] The above steps are described in detail below.

[0075] (1) Text data segmentation

[0076] The post text of each user is divided into multiple files with a week as a period, so that all the posts X of user i in the jth week are files X ij .

[0077] (2) Sentiment analysis

[0078] Stanford NLP and other analysis tools are used to analyze the sentiment of each segmented text, and each segmented text file is bound with a sentiment value Y, so that the sentiment value of user i in the jth week is Y ij .

[0079] (3) Instability evaluation

[0080] There are two ways to evaluate the emotional instability of a user, the first is the degree of change of the sentiment value, and the second is the change of the speaking frequency. Therefore, the emotional instability of a user is represented by the following formula:

[0081]

[0082]

[0083] Wherein, S i represents the emotional instability of user i, S iThe greater the value, the more unstable the user's mood, representing the user's frequent emotional outburst. n is the number of text segmentation of the user, j refers to the text of the jth week, and σ is responsible for judging whether the polarity of the adjacent two weeks is the same. Y ij+1 is the emotional value of the j+1th week, Y ij is the emotional value of the jth week, and n is the total number of periods. The formula calculates the direct emotional change difference of the user every two weeks, and when the positive and negative values of the emotion change, the change is amplified.

[0084] Then, the speech frequency change degree of a user is represented by the following formula:

[0085]

[0086] Where P represents the speech frequency change degree of the user, and the greater the P value, the greater the speech frequency change of the user, representing the user's sudden concentrated speech and sudden silence. T ij+1 is the number of posts contained in the text of the j+1th week, T ij is the number of posts contained in the text of the jth week.

[0087] (4) N value update

[0088] In the case of inconsistency between the size of the N value in the prediction result of the graph neural network and the instability represented by the above formula, it is indicated that the prediction result of the graph neural network N value at this time deviates greatly from the true situation.

[0089] Currently, the Big Five Personality Statistics Report shows that the average score of the population is: O=17.5, C=20, E=17.5, A=20, N=21. When the prediction value is higher than the average score of the population, it is judged to be a high neurotic user. When it is lower than the average score of the population, it is judged to be an emotionally stable user. Similarly, when the S value and P value of a user are greater than the average value of all users, it is determined to be a high neurotic user, and when it is lower than the average value, it is determined to be an emotionally stable user. The update formula is as follows:

[0090]

[0091] Where N new represents the adjusted prediction value of the user in neuroticism, N ori represents the prediction value of the user in neuroticism before adjustment, represents the average value of the speech frequency change degree P of all users, represents the average value of the prediction value N ori of all users in neuroticism, represents the average value of the emotional value change degree S of all users.

[0092] The embodiment of the application is a user personality prediction system based on multi-modal data fusion, comprising:

[0093] A feature vector acquisition module is configured to vectorize the text data and the picture data of the user on the network, extract a text feature vector and a picture feature vector from the text data and the picture data respectively, splice the text feature vector and the picture feature vector, and obtain a spliced vector of the user.

[0094] A prediction module is configured to use the user relationship data of the user on the social network, use the user as a node of a graph, construct edges between the nodes according to the user relationship data, use the spliced vector of the user as a semantic representation of the node, input the graph into a user personality prediction model based on a graph neural network, and output a user personality prediction value.

[0095] Further, the user personality prediction system based on multi-modal data fusion further comprises a reverse adjustment unit configured to divide the text data of the user according to a period, calculate the emotional instability degree of the user according to a pre-designed calculation formula based on the text data of different periods, and adjust the prediction value of the user according to the emotional instability degree of the user.

[0096] The feature vector acquisition module comprises a language use habit statistical unit, an avatar feature extraction unit, and a feature vector splicing unit. The language use habit statistical unit is responsible for extracting text modal features from the blog of the user. The avatar feature extraction unit is responsible for extracting image modal features from the avatar pictures used by the user. The feature vector splicing unit is responsible for combining the text and image modal features and ensuring that the feature vector dimensions of each user are the same.

[0097] The language use habit statistical unit comprises a noise removal module, a general statistical module, an LIWC statistical module, and an n-gram module, and is configured to extract text features in the blog of the user.

[0098] The avatar feature extraction unit comprises a classification pre-training unit and a classification prediction unit, and is configured to extract the classification of the image displayed by the user and use the classification as the image feature.

[0099] The feature vector splicing unit comprises a normalization unit and a combination unit, and the combination unit is configured to combine the text and image features.

[0100] The reverse adjustment unit comprises a period slicing module and an adjustment module.

[0101] The implementation method of the system is the same as the above method, and thus will not be described herein.

[0102] The application further provides a storage medium storing a computer program executable by a processor, which causes the processor to execute the steps of any of the above user personality prediction methods based on multi-modal data fusion when the computer program is run on the processor. The computer readable storage medium can include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nano system (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0103] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0104] In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0105] In several embodiments provided by the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the above-described system embodiments are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, another division mode can be used. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some service interface, system or module, and can be electrical or other forms.

[0106] The modules described as separate components can or can not be physically separate, and the components shown as modules can or can not be physical modules, i.e. can be located in one place or distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0107] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0108] The integrated module, if implemented in the form of a software function module and sold or used as an independent product, can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned memory includes: a U disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0109] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be performed by a program instructing relevant hardware, and the program can be stored in a computer readable memory, which can include a flash disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0110] The above is only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will easily think of embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present application is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional techniques in the art that are not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

[0111] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0112] Those skilled in the art readily understand that the above only describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A user personality prediction method based on multi-modal data fusion, characterized in that, The method comprises the following steps: extracting a text feature vector and a picture feature vector from the text data and the picture data of the user on the network platform, splicing the text feature vector and the picture feature vector to obtain a spliced vector of the user; using the user relationship data of the user on the network platform, taking the user as a node of a graph, constructing edges between nodes according to the user relationship data, taking the spliced vector of the user as a semantic representation of the node, inputting the graph into a user personality prediction model based on a graph neural network, and outputting a user personality prediction value; wherein, the text data of the user is divided according to a period, the emotional instability degree of the user is calculated according to the text data of different periods according to a pre-designed calculation formula, and the prediction value of the user is adjusted according to the emotional instability degree of the user; wherein, the relationship data of the user on the network platform includes only attention nodes and bidirectional attention nodes of the user, and the user personality prediction model gives different weights to the only attention nodes and the bidirectional attention nodes of the user when aggregating the relationship data between users; wherein, the text feature vector is a vector composed of the word frequency of different types of words, the picture data is a user avatar, and the picture feature vector is a vector composed of the probability that the user avatar picture belongs to different categories. 2.The user personality prediction method based on multi-modal data fusion according to claim 1, wherein, The emotional instability degree includes a sentiment value change degree and a speech frequency change degree, the sentiment value change degree of the user i is denoted as Si, and the speech frequency change degree of the user i is denoted as Pi, The calculation formula of Si and Pi is: wherein, is the sentiment value for user i in week j+1, is the sentiment value for user i in week j, n is the total number of periods, is the number of posts contained in the text for user i in week j+1, is the number of posts contained in the text for user i in week j. 3.The user personality prediction method based on multi-modal data fusion of claim 2, wherein, The calculation formula for adjusting the prediction value of the user is: wherein, represents the predicted value of neuroticism of the adjusted user i, represents the predicted value of neuroticism of the user i before adjustment, represents the mean of the degree of change in the frequency of utterance P of all users, represents the mean of the predicted value of neuroticism of all users represents the mean of the degree of change in the emotional value S of all users, represents the mean of the degree of change in the emotional value S of all users. 4.The user personality prediction method based on multi-modal data fusion of claim 1, wherein, The user personality prediction model includes five prediction sub-models, each of which is used to output a prediction value of a certain dimension of the five personality dimensions in the Big Five Personality Model, and each prediction sub-model is trained separately.

5. A user personality prediction system based on multi-modal data fusion, based on the user personality prediction method based on multi-modal data fusion according to any one of claims 1-4, characterized in that, The method comprises the following steps: a feature vector acquisition module for vectorizing the text data and the picture data of the user on the network, extracting a text feature vector and a picture feature vector from the text data and the picture data, splicing the text feature vector and the picture feature vector to obtain a spliced vector of the user; a prediction module for using the user relationship data of the user on the network, taking the user as a node of a graph, constructing edges between nodes according to the user relationship data, taking the spliced vector of the user as a semantic representation of the node, inputting the graph into a user personality prediction model based on a graph neural network, and outputting a user personality prediction value; wherein, the text data of the user is divided according to a period, the emotional instability degree of the user is calculated according to the text data of different periods according to a pre-designed calculation formula, and the prediction value of the user is adjusted according to the emotional instability degree of the user; wherein, the relationship data of the user on the network platform includes only attention nodes and bidirectional attention nodes of the user, and the user personality prediction model gives different weights to the only attention nodes and the bidirectional attention nodes of the user when aggregating the relationship data between users; wherein, the text feature vector is a vector composed of the word frequency of different types of words, the picture data is a user avatar, and the picture feature vector is a vector composed of the probability that the user avatar picture belongs to different categories. 6.The user personality prediction system based on multi-modal data fusion of claim 5, wherein, Further comprising: The reverse adjustment unit is configured to divide the text data of the user by period, calculate the emotional instability degree of the user according to the text data of different periods according to a pre-designed calculation formula, and adjust the prediction value of the user according to the emotional instability degree of the user.

7. A storage medium, characterized by The computer program is stored in the computer storage and, when the computer program is run on the processor, causes the processor to execute the steps of the method according to any one of claims 1-4.

Images