A method and system for acquiring user psychological portraits through cross-modal data fusion

Abstract

The present invention proposes a method and system for obtaining a user psychological portrait by cross-modal data fusion, the method comprising: collecting multimodal data of users; embedding the multimodal data, mapping the same embedding space, introducing a self-attention mechanism for weighted processing, and fusing the multimodal data; performing an emotional feature extraction operation on the pre-processed multimodal data, and constructing a temporal emotional model to analyze the comprehensive emotional feature vector; constructing a social relationship network according to the social relationship data of the user to fuse the social relationship data with the multimodal data; revising the user psychological portrait based on the judgment result and then fusing the user psychological portrait of the social relationship data; defining a reinforcement learning model to adaptively optimize the user psychological portrait. The present invention can not only realize the efficient fusion of multimodal data and the generation of dynamic psychological portraits, but also ensure the accuracy and real-time performance of the system in practical applications by introducing cutting-edge technologies such as reinforcement learning and graph neural networks.

Description

Technical Field

[0001] The present invention belongs to the field of user psychological profiling, and in particular, relates to a method and system for obtaining user psychological profiling by cross-modal data fusion. Background Art

[0002] In the modern information society, with the popularization of Internet technology and smart devices, users' activities on the Internet are becoming more and more diversified, generating a huge amount of multimodal data, including text, images, audio, video and behavioral data. These multimodal data not only reflect the user's behavior and interests, but also contain rich psychological state information. Therefore, it is of great application value to construct a user's psychological portrait by effectively fusing and analyzing these multimodal data. However, the existing user portrait technology mainly focuses on the processing of single modal data, such as text-based sentiment analysis, image-based facial expression recognition, etc., lacking the comprehensive use of multimodal data, resulting in its incomplete and inaccurate portrayal of the user's psychological state.

[0003] At present, some studies have attempted to use multimodal data fusion to obtain user psychological portraits, but there are still many problems. First, the heterogeneity and diversity of multimodal data make it difficult to effectively fuse different modal data.

[0004] Existing methods mostly use simple feature concatenation or weighted averaging for cross-modal data fusion, which cannot fully utilize the complementary information of different modal data, resulting in poor fusion effect.

[0005] Secondly, existing methods mainly rely on static models for sentiment analysis and psychological state prediction, and cannot dynamically capture changes in the user's psychological state. However, the user's psychological state is time-varying and complex, and static models are difficult to adapt to such dynamic changes. Thirdly, social relationships play an important role in the user's psychological state, but existing methods rarely consider the user's social network data and ignore the impact of the user's interaction with others on his or her psychological state.

[0006] Furthermore, the social scene also affects the user's real psychological state, making it more difficult to obtain and update user portraits. When people are in some sensitive scenes, such as meetings or public places, their inner psychological portraits will be affected by the scene, causing deviations in the psychological portraits extracted from the face and language, resulting in inaccurate psychological portraits.

[0007] Finally, the data processing and model training process lacks an adaptive optimization mechanism, making it difficult to make dynamic adjustments based on users' real-time behavior and feedback, resulting in less than ideal performance of the model in practical applications.

[0008] Therefore, there is an urgent need for a more advanced multimodal data fusion method that can effectively fuse and mine potential correlations in multimodal data while maintaining data integrity, thereby enhancing the accuracy and comprehensiveness of user psychological portraits. Summary of the invention

[0009] The purpose of this invention is to propose a method for obtaining user psychological portraits through cross-modal data fusion, which constructs a more comprehensive, accurate and dynamic user psychological portrait through comprehensive utilization and adaptive optimization of multimodal data, thereby overcoming the limitations of the existing technology.

[0010] In order to achieve the above object, a first aspect of the present invention provides a method for obtaining a user psychological portrait by cross-modal data fusion, the method comprising:

[0011] S1. Collect multimodal data of users and pre-process the data; the multimodal data includes text data, image data, behavior data and social relationship data;

[0012] S2. Embed the preprocessed multimodal data, use a cross-modal alignment network to map the multimodal data to the same embedding space, introduce a self-attention mechanism to embed the features in the space for weighted processing, fuse the weighted multimodal data, and obtain multimodal fusion features;

[0013] S3, perform sentiment feature extraction on the multimodal fusion features, and perform cross-modal sentiment feature fusion on the extracted sentiment features to obtain a comprehensive sentiment feature vector, and construct a time series sentiment model to analyze the comprehensive sentiment feature vector to capture sentiment change trends;

[0014] S4. Based on the user's social relationship data, a social relationship network is constructed to integrate the social relationship data with multimodal data, analyze the impact of social relationships on the user's psychological state, and obtain a user psychological portrait that integrates the social relationship data; wherein the social relationship data includes friend relationships, interaction frequency, and message exchange records;

[0015] S5. Collect the user's situational information to design a scenario-sensitive score, judge the social environment based on the scenario-sensitive score, and correct the user's psychological profile based on the judgment result and the user's psychological profile integrated with the social relationship data to obtain the final psychological profile correction objective function;

[0016] S6. According to the final psychological portrait correction objective function, a reinforcement learning model is defined to adaptively optimize the user psychological portrait and update the user psychological portrait in real time.

[0017] Further, the preprocessing includes data cleaning, data standardization, and data synchronization and alignment;

[0018] The data cleaning includes cleaning text data, denoising image data, denoising audio data, and deduplication, completion, and consistency checking of behavior data and social relationship data.

[0019] The data standardization processing includes word vectorization of text data, normalization of image data, feature extraction of audio data and standardization of behavior data;

[0020] The data synchronization and alignment synchronizes the multimodal data by timestamp, aligns them to the same time axis, and uses linear interpolation to complete the missing data.

[0021] Furthermore, the use of linear interpolation to complete the missing data is expressed as follows:

[0022]

[0023] Among them, t1 and t2 represent adjacent time points, D sync (t) represents the interpolated data, D sync (t1) represents the data at t1, D sync (t2) represents the data at time t2.

[0024] Furthermore, for text data, the BERT model is used for vector embedding and sentiment feature extraction; for image data, the ResNet50 model is used for vector embedding and sentiment feature extraction; for audio data, the VGGish model is used for vector embedding and sentiment feature extraction.

[0025] Furthermore, a cross-modal alignment network f is used based on the features extracted from text data, image data, or audio data. CMAN Mapping data of different modalities to the same embedding space is expressed as follows:

[0026] v aligned (x i ) = f CMAN (v t (x i ),v i (x i ),v a (x i ))

[0027] Among them, x i represents text data, image data, or audio data, f CMAN represents the cross-modal alignment network, v aligned (x i ) represents the feature vector mapped to the same embedding space, v t 、v iand v a Represent the embedding vectors of text, image and audio respectively;

[0028] The self-attention mechanism is used to weight the features in the embedding space, which can be expressed as follows:

[0029] v att (x i )=W att ·v aligned (x i )

[0030] Among them, W att represents the self-attention weight matrix obtained through training, v att Represents the feature vector after processing by the self-attention mechanism;

[0031] Adding regularization terms improves the generalization ability and robustness of the model, as shown below:

[0032]

[0033] Among them, R1(W) represents the regularization term of multimodal data, ∥W att ∥ F represents the Frobenius norm, v orig (x i ) represents the original feature vector, λ1 and λ2 are the weight coefficients of the regularization term respectively;

[0034] Then the overall loss function L1 of multimodal data is expressed as follows:

[0035]

[0036] in, represents the alignment loss, which is used to measure the effect of feature alignment between different modalities;

[0037] The weighted feature vectors of text, image, and audio processed by the self-attention mechanism are fused to generate a comprehensive multimodal feature representation, which is expressed as follows:

[0038] v fusion (x i ) = f fusion (v att (x i ))

[0039] Among them, v fusion (x i ) represents the final fused multimodal feature vector, f fusion represents the fusion function;

[0040] Using cross-modal sentiment fusion network f CMEFNThe emotional features extracted from the weighted feature vectors of text, image and audio are fused into a comprehensive emotional feature vector e fusion (x i ), which is expressed as follows:

[0041] e fusion (x i ) = f CMEFN (e t (x i ),e i (x i ),e a (x i ))

[0042] Among them, e t 、e i and e a Respectively represent the sentiment features extracted from the weighted feature vectors of text, image and audio.

[0043] Furthermore, according to the comprehensive emotional feature vector e fusion (x i ) uses enhanced LSTM to build a time series emotion model and provide a dynamic emotional state representation, which is expressed as follows:

[0044] s t =f E-LSTM (e fusion (x i:t ))+δ

[0045] Among them, e fusion (x i:t ) represents the sentiment feature vector in time series t, s t represents the emotional state vector of the current time step, δ is used to smooth emotional changes and prevent excessive emotional fluctuations, and f E-LSTM Represents the enhanced LSTM model function;

[0046] Among them, in the process of emotional feature fusion and time series modeling, the emotional similarity regularization term R is added sim and the time series smoothing regularization term R smooth , the total regularization term R2(W) of the sentiment feature is obtained as:

[0047]

[0048] in, Used to maintain the similarity of the same user's emotional characteristics, Used to smooth the temporal emotional state, λ3 and λ4 are the weight coefficients of the total regularization term R2(W) of the emotional feature;

[0049] The overall loss function L2 with emotional features is expressed as follows:

[0050]

[0051] in, Represents the loss of emotion recognition and temporal modeling, which is used to measure the effect of emotion feature extraction and temporal modeling.

[0052] Furthermore, the S4 specifically includes:

[0053] S401: construct a social relationship network based on social relationship data, wherein the social relationship network is represented as a graph structure in, represents the set of user nodes, ε represents the set of edges between users, and the weight of the edge is w ij It is determined by the interaction frequency and message exchange volume between user i and user j, which can be expressed as follows:

[0054]

[0055] Among them, f ij represents the interaction frequency between user i and user j, m ij represents the amount of message exchanges, α and β are weight coefficients, controlling the impact of interaction frequency and message traffic flow;

[0056] S402. Each user node Contains the user's multimodal emotional feature vector e fusion (x i ) and social relationship feature vector r i , forming the node feature vector h i ;

[0057] S403: Use GCN to extract features and propagate information on social networks. By superimposing multiple layers of GCN, the aggregation and information propagation of social network node features are achieved, and finally a high-dimensional feature representation of each node is obtained. It is expressed as follows:

[0058]

[0059] Among them, H (l) is the feature matrix of the l-th layer node, A is the normalized adjacency matrix, W (l) is the weight matrix of the lth layer, σ(·) is the activation function, H (l+1) is the node feature matrix of the l+1th layer;

[0060] S404. Adding social relationship consistency regularization term to GCN and feature similarity regularization term The total regularization term is for:

[0061]

[0062] in, Used to maintain the consistency of social relationship nodes, Used to maintain feature similarity, λ5 and λ6 are the weight coefficients of the regularization term, represents the high-dimensional feature representation i, Represents high-dimensional feature representation j, h i represents low-dimensional feature representation;

[0063] The overall loss function L GCN-total for:

[0064]

[0065] in, Represents the loss of GCN, which is used to measure the effect of feature extraction and information propagation;

[0066] S405. Represent the high-dimensional features output by GCN and the user's emotional state vector s t Fusion, generate the user's comprehensive portrait feature vector p i , expressed as:

[0067]

[0068] Among them, f fuse is a fusion function used to combine high-dimensional features and emotional states.

[0069] Furthermore, the correction of the user psychological portrait based on the judgment result and the integration of the user psychological portrait of the social relationship data includes a segmented correction mechanism and the introduction of a situational sensitivity weight, and the segmented correction mechanism includes linear adjustment and nonlinear adjustment;

[0070] The scenario information includes geographic location, ambient noise level, and device usage; the social environment is judged based on the scenario sensitivity score, as shown below:

[0071] S scene =f scene (c i )=σ(W s ·c i +b s )

[0072] Among them, f scene represents a deep neural network, including multiple hidden layers and an output layer; W s represents the weight matrix, b s represents the bias vector, σ represents the activation function, and the output Sscene The value range of c is [0,1], where values ​​close to 1 represent sensitive scenarios and values ​​close to 0 represent non-sensitive scenarios. i Represents the scene feature vector, denoted as c i =[L i ; T i ; N i ;D i ], where L i The code representing the position, T i The code representing time, N i The code representing the noise level, D i A code indicating the usage of the device.

[0073] Furthermore, the linear adjustment directly affects the emotional dimension of scene sensitivity, and is expressed as follows:

[0074] e adjusted,k (x i )=e fusion,k (x i )×(1-β k ·S scene )

[0075] Among them, e fusion,k (x i ) represents the original score of the kth sentiment dimension, S scene represents the scene sensitivity score, β k Represents the coefficient used to control the adjustment strength of this dimension;

[0076] The nonlinear adjustment has a nonlinear dependence on the emotion dimension of scene sensitivity, and adopts exponential adjustment, which is expressed as follows:

[0077] e adjusted,k (x i )=e fusion,k (x i )×exp(-γ k ·S scene )

[0078] Among them, γ k represents the adjustment coefficient, which determines the intensity of the response of this dimension to the scene sensitivity;

[0079] The scenario sensitivity weight represents the sensitivity of each emotional dimension to the scenario, which is expressed as follows:

[0080] e adjusted (x i )=e fusion (x i )×(1-ψ⊙S scene )

[0081] Among them, ψ represents the scenario sensitivity weight vector, which indicates the sensitivity of each emotional dimension; ⊙ is the Hadamard product, which is multiplied element by element. scene Rate the sensitivity of the scene.

[0082] In a second aspect of the present invention, a user psychological portrait acquisition system for cross-modal data fusion is provided, the system comprising:

[0083] A multimodal data acquisition unit, used to collect multimodal data of the user and pre-process the data; the multimodal data includes text data, image data, behavior data and social relationship data;

[0084] The vector embedding unit is used to embed the preprocessed multimodal data. The cross-modal alignment network is used to map the multimodal data to the same embedding space. The self-attention mechanism is introduced to perform weighted processing on the features in the embedding space. The weighted multimodal data is fused to obtain multimodal fusion features.

[0085] The emotion feature extraction unit is used to extract emotion features from the pre-processed multi-modal data, fuse the extracted emotion features with cross-modal emotion features, obtain a comprehensive emotion feature vector, and construct a time series emotion model to analyze the comprehensive emotion feature vector and capture the emotion change trend.

[0086] A user psychological portrait construction unit is used to construct a social relationship network based on the user's social relationship data, integrate the social relationship data with multimodal data, analyze the impact of social relationships on the user's psychological state, and obtain a user psychological portrait that integrates the social relationship data; wherein the social relationship data includes friend relationships, interaction frequency, and message exchange records;

[0087] A user psychological portrait correction unit is used to collect the user's situational information to design a scenario sensitivity score, judge the social environment according to the scenario sensitivity score, and correct the user psychological portrait based on the judgment result and the user psychological portrait of the social relationship data to obtain a final psychological portrait correction objective function; wherein the situational information includes geographic location, environmental noise level and device usage;

[0088] The user psychological portrait updating unit is used to define a reinforcement learning model to adaptively optimize the user psychological portrait and update the user psychological portrait in real time.

[0089] The beneficial technical effects of the present invention are at least as follows:

[0090] (1) The present invention introduces cross-modal embedding technology to map data of different modalities such as text, image, and audio into the same embedding space to achieve seamless fusion. In this way, the complementary information of data of different modalities can be fully utilized, overcoming the shortcomings of existing methods in cross-modal data fusion, thereby generating a more comprehensive user psychological portrait.

[0091] (2) The present invention adopts a time series model to model the user's emotional data at different time points and capture the trend of emotional changes. Compared with the static model, the time series model can dynamically reflect the changes in the user's psychological state and provide a more accurate psychological portrait, solving the problem that the existing method cannot dynamically capture the changes in the user's psychological state.

[0092] (3) The present invention uses graph neural networks to analyze users’ social relationship data and integrates social network data with other modal data. In this way, the impact of users’ interactions with others on their psychological state can be captured, the dimension and accuracy of psychological portraits can be enhanced, and the problem of existing methods ignoring social relationship data can be compensated.

[0093] (4) The present invention introduces a reinforcement learning algorithm to optimize the cross-modal data fusion process and adjusts the data processing strategy according to the user's real-time feedback and behavior. The reinforcement learning mechanism enables the system to adaptively optimize the data collection and processing process and achieve real-time updating of the user's psychological portrait, overcoming the problem that the existing methods lack an adaptive optimization mechanism.

[0094] (5) The present invention combines the user's geographic location, time, environment and other situational information to enhance the contextual awareness of the data. Through situational awareness technology, the system can better understand the user's psychological state and its changing background, thereby providing a more personalized and context-related psychological portrait, solving the shortcomings of existing methods in situational information processing.

[0095] (6) The present invention not only realizes efficient fusion of multimodal data and generation of dynamic psychological portraits, but also ensures the accuracy and real-time performance of the system in practical applications by introducing cutting-edge technologies such as reinforcement learning and graph neural networks. These innovations respectively address the main problems in the prior art and provide systematic and comprehensive solutions, making the acquisition of user psychological portraits more scientific and practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0097] Figure 1 This is a flow chart of a method for acquiring user psychological portraits through cross-modal data fusion according to the present invention. DETAILED DESCRIPTION

[0098] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0099] like Figure 1 As shown, a method for obtaining a user psychological portrait by cross-modal data fusion provided by an embodiment of the present invention includes the following steps:

[0100] S1. Collect multimodal data of users and pre-process the data; the multimodal data includes text data, image data, behavior data and social relationship data.

[0101] Specifically, users' multimodal data is collected, including text data (such as social media posts, comments, etc.), image data (such as uploaded photos, pictures, etc.), audio data (such as voice messages, voice calls, etc.), behavioral data (such as clicks, browsing history, geographic location information, etc.), and social relationship data (such as friend relationships, interaction frequency, etc.).

[0102] Furthermore, API interfaces or web crawler technologies are used to collect data to ensure the diversity and breadth of data sources.

[0103] Furthermore, the text data is cleaned to remove stop words, punctuation marks and special characters, and stem extraction and word segmentation are performed.

[0104] Denoise the image data using a Gaussian filter

[0105]

[0106] Where σ represents the standard deviation.

[0107] The audio data is denoised and the background noise is eliminated by spectral subtraction. The specific formula is:

[0108]

[0109] in, represents the denoised spectrum, α represents the noise suppression coefficient, |X(t,f)| and |N(t,f)| represent the original audio spectrum and the noise spectrum, respectively.

[0110] De-duplicate, complete, and check the consistency of behavioral data and social relationship data to ensure data integrity and accuracy.

[0111] Furthermore, the data standardization process is as follows:

[0112] The text data is vectorized and the Word2Vec or GloVe model is used to map the words into the vector space. Assume that the word w i The word vector is represented as v(w i ).

[0113] Normalize the image data and normalize the pixel values ​​to the [0,1] interval. The specific formula is:

[0114]

[0115] Among them, I represents the original image, I norm represents the normalized image, min(I) and max(I) represent the minimum and maximum pixel values ​​of the image, respectively.

[0116] The audio data is feature extracted and the MFCC (Mel Frequency Cepstral Coefficient) method is used to convert the audio signal into a feature vector. Assuming the audio signal is A(t), its MFCC is represented by M(A).

[0117] The behavioral data were standardized and converted into a standard normal distribution with a mean and unit variance. The specific formula is:

[0118]

[0119] Among them, x represents the original data, μ and σ represent the mean and standard deviation of the data respectively, and z represents the standardized data.

[0120] Furthermore, the data synchronization and alignment are:

[0121] The multimodal data are synchronized by timestamp and aligned to the same time axis. Assume that each modal data is D t , D i , D a , respectively represent text, image and audio data, and the aligned multimodal data is represented as D sync (t).

[0122] Linear interpolation is used to complete the missing data. The specific formula is:

[0123]

[0124] Among them, t1 and t2 represent adjacent time points, D sync (t) represents the interpolated data.

[0125] S2. Embed the preprocessed multimodal data, use the cross-modal alignment network to map the multimodal data to the same embedding space, introduce the self-attention mechanism to weight the features in the embedding space, and fuse the weighted multimodal data to obtain multimodal fusion features.

[0126] Specifically, for text data, the pre-trained BERT model is used to embed the sentence into a vector v t (s i ), where s i Represents the i-th sentence.

[0127] v t (s i )=BERT(s i )

[0128] Through the pre-training of the BERT model, text data can be converted into high-dimensional vector representations containing rich semantic information.

[0129] For image data, we use the pre-trained ResNet50 model to transform image I i Embedded as vector v i (I i ).

[0130] v i (I i )=ResNet50(I i )

[0131] The ResNet50 model has a deep convolutional structure that can extract high-level features from images.

[0132] For audio data, use the pre-trained VGGish model to convert the audio signal A i Embedded as vector v a (A i ).

[0133] v a (A i )=VGGish(A i )

[0134] The VGGish model processes the spectral features of the audio signal and converts it into an audio feature vector.

[0135] In addition, in order to map data of different modalities to the same embedding space, a cross-modal alignment network (CMAN) is used. Define the cross-modal alignment network f CMAN , mapping the feature vectors of text, images, and audio into the same embedding space.

[0136] v aligned (x i ) = f CMAN (v t (x i ),v i (x i ),v a (x i ))

[0137] Among them, x i It can be text, image or audio data. aligned (x i ) represents the feature vector mapped to the same embedding space.

[0138] CMAN achieves seamless alignment and fusion of data by learning the relationship between different modalities, ensuring that different modal features have the same representation capabilities in the same space.

[0139] Furthermore, the self-attention mechanism is used to weight the features in the embedding space to extract important information. Define the self-attention weight matrix W att and the eigenvector v att Relationship:

[0140] v att (x i )=W att ·v aligned (x i )

[0141] Among them, W att is the weight matrix obtained through training, v att (x i ) is the feature vector after being processed by the self-attention mechanism.

[0142] The self-attention mechanism can dynamically adjust the weights according to the importance of the features, improve the accuracy of feature representation, and extract information that is more important to the user's psychological portrait.

[0143] Furthermore, in the cross-modal embedding process, in order to improve the generalization ability and robustness of the model, a regularization term is added to improve the generalization ability and robustness of the model, as shown below:

[0144]

[0145] Among them, R1(W) represents the regularization term of multimodal data, ∥W att ∥ F represents the Frobenius norm, v orig (x i) represents the original feature vector, λ1 and λ2 are the weight coefficients of the regularization term, respectively; the regularization term includes the Frobenius norm and feature alignment error, which ensures the generalization ability and robustness of the model during training.

[0146] Then the overall loss function L1 of multimodal data is expressed as follows:

[0147]

[0148] in, represents the alignment loss, which is used to measure the effect of feature alignment between different modalities;

[0149] The weighted feature vectors of text, image, and audio processed by the self-attention mechanism are fused to generate a comprehensive multimodal feature representation, which is expressed as follows:

[0150] v fusion (x i ) = f fusion (v att (x i ))

[0151] Among them, v fusion (x i ) represents the final fused multimodal feature vector, f fusion Represents the fusion function. The fused feature vector contains the complementary information of multimodal data, providing a rich feature basis for subsequent sentiment analysis, social relationship analysis and adaptive optimization.

[0152] S3. Perform sentiment feature extraction on the fused multimodal feature vector, and perform cross-modal sentiment feature fusion on the extracted sentiment features to obtain a comprehensive sentiment feature vector. Then, construct a time series sentiment model to analyze the comprehensive sentiment feature vector and capture the sentiment change trend.

[0153] Among them, it is necessary to dynamically capture the user's emotional changes and provide real-time emotional analysis and prediction.

[0154] Specifically, use the fused feature vector v in step S2 fusion (x i ), and further through the sentiment analysis model f emotion Extract emotional features fusion (x i ):

[0155] e fusion (x i ) = f emotion (v fusion (x i ))=σ(W e ·v fusion (xi )+b e )

[0156] Among them, W e is the weight matrix, b e is the bias vector, and σ is the ReLu activation function. This sentiment analysis model can extract sentiment-related information from the fused features.

[0157] Then, the emotion feature fusion network (EFFN) is used to further fuse the above emotion features to generate the final emotion feature vector e fusion (x i ):

[0158] e fusion (x i ) = f EFFN (e fusion (x i ))=Attention(W f ·e fusion (x i )+b f )

[0159] Among them, W f is the weight matrix of the fusion layer, b f is the bias vector, and Attention represents the attention mechanism. This fusion network performs weighted fusion of different emotional features through the attention mechanism to ensure the seamless integration and comprehensiveness of emotional features.

[0160] Furthermore, Enhanced LSTM (E-LSTM) is used to model the time series sentiment data to capture the trend of sentiment changes. E-LSTM adds a sentiment change smoothing term δ to the traditional LSTM. The formula is:

[0161] s t =f E-LSTM (e fusion (x i:t ))+δ

[0162] Among them, e fusion (x i:t ) represents the sentiment feature vector in time series t, s t Represents the emotional state vector of the current time step, and δ is used to smooth emotional changes and prevent excessive emotional fluctuations.

[0163] Through the memory and forgetting mechanism and the emotion change smoothing term, E-LSTM can effectively capture the long-range dependencies of emotion changes and provide dynamic emotion state representation.

[0164] In the process of sentiment feature fusion and time series modeling, the sentiment similarity regularization term R is added sim and the time series smoothing regularization term R smooth , the total regularization term R2(W) of the sentiment feature is obtained as:

[0165]

[0166] in, Used to maintain the similarity of the same user's emotional characteristics, Used to smooth the temporal emotional state, λ3 and λ4 are the weight coefficients of the total regularization term R2(W) of the emotional feature;

[0167] The overall loss function L2 with emotional features is expressed as follows:

[0168]

[0169] in, Represents the loss of emotion recognition and temporal modeling, which is used to measure the effect of emotion feature extraction and temporal modeling.

[0170] Through the above detailed steps, the extraction and fusion of multimodal emotional features are ensured, the trend of emotional changes is captured by the time series model, and the stability and generalization ability of the model are optimized through the regularization term and loss function. The design of each step is closely connected and serves the overall goal of the patent together, solving the problems of incomplete and insufficient dynamics of emotional analysis in the existing technology, and providing a solid foundation for the accuracy and real-time performance of user psychological portraits.

[0171] S4. Based on the user's social relationship data, a social relationship network is constructed to integrate the social relationship data with multimodal data, analyze the impact of social relationships on the user's psychological state, and obtain a user psychological portrait that integrates the social relationship data; wherein the social relationship data includes friend relationships, interaction frequency, and message exchange records. Specifically including:

[0172] S401: construct a social relationship network based on social relationship data, wherein the social relationship network is represented as a graph structure in, represents the set of user nodes, ε represents the set of edges between users, and the weight of the edge is w ij It is determined by the interaction frequency and message exchange volume between user i and user j, which can be expressed as follows:

[0173] w ij =α·f ij +β·m ij

[0174] Among them, f ij represents the interaction frequency between user i and user j, mij represents the amount of message exchanges, α and β are weight coefficients, controlling the impact of interaction frequency and message traffic flow;

[0175] S402. Each user node Contains the user's multimodal emotional feature vector e fusion (x i ) and social relationship feature vector r i , forming the node feature vector h i , expressed as:

[0176] h i =[e fusion (x i );r i ]

[0177] Among them, [·;·] represents the vector concatenation operation, e fusion (x i ) is the multimodal sentiment feature vector, r i is the social relationship feature vector;

[0178] S403: Use GCN to extract features and propagate information on social networks. By superimposing multiple layers of GCN, the aggregation and information propagation of social network node features are achieved, and finally a high-dimensional feature representation of each node is obtained. It is expressed as follows:

[0179] H (l+1) =σ(AH (l) W (l) )

[0180] Among them, H (l) is the feature matrix of the l-th layer node, A is the normalized adjacency matrix, W (l) is the weight matrix of the lth layer, σ(·) is the activation function, H (l+1) is the node feature matrix of the l+1th layer; by superimposing multiple layers of GCN, the aggregation and information propagation of social network node features are realized, and finally the high-dimensional feature representation of each node is obtained

[0181] S404. Adding social relationship consistency regularization term to GCN and feature similarity regularization term The total regularization term is for:

[0182]

[0183] in, Used to maintain the consistency of social relationship nodes, Used to maintain feature similarity, λ5 and λ6 are the weight coefficients of the regularization term, represents the high-dimensional feature representation i, Represents high-dimensional feature representation j, h i represents low-dimensional feature representation;

[0184] The overall loss function L GCN-total for:

[0185]

[0186] in, Represents the loss of GCN, which is used to measure the effect of feature extraction and information propagation;

[0187] S405. Represent the high-dimensional features output by GCN and the user's emotional state vector s t Fusion, generate the user's comprehensive portrait feature vector p i , expressed as:

[0188]

[0189] Among them, f fuse is a fusion function used to combine high-dimensional features and emotional states.

[0190] The final comprehensive portrait feature vector p i It includes the user's emotional state and social relationship information, providing strong support for the accuracy and comprehensiveness of the user's psychological portrait.

[0191] S5. Collect the user's situational information to design a scenario-sensitive score, judge the social environment based on the scenario-sensitive score, and correct the user's psychological portrait based on the judgment result and the user's psychological portrait integrated with the social relationship data to obtain the final psychological portrait correction objective function.

[0192] Specifically, further collect the user's current context information, including but not limited to:

[0193] Location i The geographic location obtained through GPS or network location information, such as office (L i =[0,1,0,0]), home (L i =[1,0,0,0]), public places (L i =[0,0,1,0]), etc., encoded as vector L i .

[0194] Time information T i Include specific time points (such as 10 a.m.), weekends or weekdays (such as weekdays are T i= [0,1], weekend is T i =[1,0]), etc. The time information obtained by the timestamp is encoded as a vector T i .

[0195] Ambient noise level N i The microphone collects the surrounding noise level in real time, such as 40dB (N i =[0.4]), 60dB(N i =[0.6]), etc., and use the audio analysis model to convert the noise level (such as dB) into a numerical feature and encode it into a vector N i .

[0196] Equipment usage D i Current device usage, such as using office software (D i =[1,0]), social media (D i =[0,1]), etc., encoded as vector D i .

[0197] Furthermore, these features are combined into a context feature vector c i :

[0198] c i =[L i ; T i ; N i ;D i ]

[0199] Each of these vectors represents a specific physical meaning, such as L i The code representing the position, T i The code representing time, N i The code representing the noise level, D i Codes representing device usage. All vector dimensions are normalized to ensure their comparability and model validity.

[0200] Furthermore, we construct a scene classification function f scene :

[0201] Specifically, scene It is a deep neural network (DNN) whose structure includes multiple hidden layers and an output layer. The input is the scene feature vector c i , the output is the scene sensitivity score S scene :

[0202] S scene =f scene (c i )=σ(W s ·c i +b s)

[0203] Among them, W s is the weight matrix, b s is the bias vector, σ is the activation function (such as the Sigmoid function), and the output S scene The value range is [0,1], where values ​​close to 1 indicate sensitive scenarios and values ​​close to 0 indicate non-sensitive scenarios.

[0204] Sensitive scenarios usually involve scenarios where users need to concentrate highly or participate in social interactions, such as:

[0205] Meeting: The user is in an office or conference room during work hours, with low ambient noise, and the device is using office applications.

[0206] Public speaking: The user is in a public place during the day with a high ambient volume, indicating that the user may be giving a speech or making a public speech.

[0207] Non-sensitive scenarios are usually scenarios where users are in a state of strong personal privacy or relaxation, such as:

[0208] Home scenario: The user is at home after get off work or on the weekend in a quiet environment, and the device is using entertainment applications.

[0209] Leisure scenario: Users are in parks or at home, using social media or entertainment apps during non-working hours.

[0210] Understandably, in order to enhance the generalization ability of the model and avoid overfitting, a penalty term based on L2 regularization is introduced:

[0211]

[0212] Among them, λ s is the weight coefficient of the regularization term, which is used to control the influence of regularization. The overall loss function of scene classification is:

[0213]

[0214] in, is the classification loss, usually using cross entropy loss.

[0215] Furthermore, in step S4, the comprehensive emotional feature vector e is obtained. fusion (x i ), which reflects the multimodal emotional characteristics of the user. In S5, the scene sensitivity score S scene This feature vector is modified to more accurately reflect the user's psychological state in a specific situation.

[0216] Specifically, the correction function fadjust Score based on scene sensitivity scene , for the sentiment feature vector e fusion (x i ) to make adjustments:

[0217] e adjusted (x i ) = f adjust (e fusion (x i ),S scene )

[0218] Among them, e adjusted (x i ) is the corrected sentiment feature vector, e fusion (x i ) is the original sentiment feature vector obtained from the fourth step.

[0219] Furthermore, a segmented correction mechanism is designed to design different adjustment strategies for different emotional dimensions (such as pleasure, anxiety, etc.), including:

[0220] (1) Linear adjustment: Linear adjustment is used for the emotional dimension that has a direct impact on scene sensitivity:

[0221] e adjusted,k (x i )=e fusion,k (x i )×(1-β k ·S scene )

[0222] Among them, e fusion,k (x i ) is the original score of the kth sentiment dimension, S scene is the scene sensitivity score, β k is a coefficient used to control the strength of the adjustment of this dimension.

[0223] (2) Non-linear adjustment: For the emotional dimension with non-linear dependence on scene sensitivity, exponential adjustment is adopted:

[0224] e adjusted,k (x i )=e fusion,k (x i )×exp(-γ k ·S scene )

[0225] Among them, γ k is the adjustment coefficient, which determines the response strength of this dimension to the scene sensitivity.

[0226] Specifically, for example, suppose W s=[0.5,0.3,0.7,0.2,0.4,0.6],b s =0.2, c i =[1,0,0,0,0.6,1,0.4,1] (corresponding to the scenario of home, working day, 60dB noise, using office software), the calculation results are:

[0227] S scene =σ([0.5,0.3,0.7,0.2,0.4,0.6]·[1,0,0,0,0.6,1,0.4,1]+0.2)

[0228] =σ(0.5·1+0.6·0.7+0.4·0.6+0.6·1+0.2)=σ(1.68)

[0229] Among them, σ(1.68)≈0.843, indicating that the current scene is a relatively sensitive scene.

[0230] Assume that in a meeting scenario, the user's happiness e fusion,joy (x i )=0.8, scene sensitivity score S scene =0.9, adjustment coefficient β joy =0.3, then the adjusted pleasure is:

[0231] e adjusted,joy (x i )=0.8×(1-0.3×0.9)=0.8×0.73=0.584

[0232] For anxiety fusion,anxiety (x i )=0.5, assuming the scene sensitivity score S scene =0.7, adjustment coefficient γ anxiety =0.8, then the adjusted anxiety level is:

[0233] e adjusted,anxiety (x i )=0.5×exp(-0.8×0.7)=0.5×exp(-0.56)≈0.5×0.571=0.2855.

[0234] Furthermore, the introduction of the context sensitivity weight ψ is to achieve more refined adjustment. The context sensitivity weight vector ψ is defined to represent the sensitivity of each emotional dimension to the scene:

[0235] e adjusted (x i )=e fusion (x i )×(1-ψ⊙S scene )

[0236] Among them, ψ is a vector representing the sensitivity of each emotional dimension, ⊙ is the Hadamard product (element-by-element multiplication), S scene Rate the sensitivity of the scene.

[0237] It is understandable that the corrected emotional feature vector e adjusted (x i ) and existing user psychological portraits initial (x i ) fusion to generate the final user psychological portrait p final (x i ):

[0238]

[0239] Among them, p initial (x i ) is the user psychological portrait before correction, e adjusted (x i ) is the emotion feature vector after scene correction, Represents a fusion operation, such as weighted average or simple addition.

[0240] Furthermore, in the process of updating the portrait, in order to avoid excessive mutations in the psychological portrait, a smoothing regularization term is used Control the smoothness of image changes:

[0241]

[0242] Among them, λ smooth It is the coefficient that controls the correction range to ensure that the image adjustment is natural and reasonable.

[0243] The final psychological portrait correction objective function:

[0244]

[0245] in, It represents the matching loss between the corrected emotional features and the current psychological portrait. By optimizing this objective function, we can ensure that the user's psychological portrait is reasonably updated when the scene changes.

[0246] To further illustrate, in step S4, the user's comprehensive emotional feature vector e is obtained through multimodal data fusion. fusion (x i ), and based on this generated a preliminary psychological portrait p initial (x i ). Assume that p initial (x i ) contains scores for the following sentiment dimensions:

[0247] Pleasure: e fusion,joy (x i )=0.75;

[0248] Anxiety level: e fusion,anxiety (x i )=0.3;

[0249] Excitement: e fusion,excitement (x i )=0.65;

[0250] Preliminary psychological portrait initial (x i ) is represented as follows:

[0251] p initial (x i )=[0.75,0.3,0.65]

[0252] Through the scene correction in S5, the emotional features in S4 are adjusted. Assuming that the current scene is a highly sensitive meeting scene, the scene sensitivity score S scene =0.8. According to the correction function f adjust , the adjusted sentiment features are as follows:

[0253] Pleasure: Through linear adjustment, e adjusted,joy (x i )=0.75×(1-0.3×0.8)=0.75×0.76=0.57;

[0254] Anxiety: Through nonlinear adjustment, e adjusted,anxiety (x i )=0.3×exp(-0.5×0.8)=0.3×exp(-0.4)≈0.3×0.6703=0.2011;

[0255] Excitement: Assume that the context sensitivity weight ψ excitement =0.4 for correction, e adjusted,excitement (x i ) = 0.65 × (1-0.4 × 0.8) = 0.65 × 0.68 = 0.442;

[0256] Corrected emotional feature vector e adjusted (x i ) is expressed as follows:

[0257] e adjusted (x i )=[0.57,0.2011,0.442]

[0258] The corrected emotional feature vector e adjusted (xi ) and preliminary psychological portrait p initial (x i ) to generate the final user psychological portrait p final (x i ). Assuming that the weighted average fusion method is adopted, the weight of the preliminary portrait is 0.6, and the weight of the corrected emotional feature is 0.4, the final fused portrait is:

[0259] p final (x i )=0.6×p initial (x i )+0.4×e adjusted (x i )

[0260] For pleasure:

[0261] p final,joy (x i )=0.6×0.75+0.4×0.57=0.45+0.228=0.678

[0262] Regarding anxiety:

[0263] p final,anxiety (x i )=0.6×0.3+0.4×0.2011=0.18+0.08044=0.26044

[0264] For Fendu:

[0265] p final,excitement (x i )=0.6×0.65+0.4×0.442=0.39+0.1768=0.5668

[0266] The final generated user psychological portrait p final (x i ) is expressed as follows:

[0267] p final (x i )=[0.678,0.26044,0.5668]

[0268] In the process of updating the portrait, in order to avoid excessive mutation of the psychological portrait, a smoothing regularization term is used. Control the smoothness of the change of the profile. Here, the purpose of regularization is to prevent overfitting in the model or data processing process, that is, the model is too dependent on specific data or conditions, resulting in poor performance when facing new data. In our scenario, the goal of smooth regularization is to prevent sudden changes in the update of the psychological profile and ensure that the change between the user psychological profile after each adjustment and the original profile is reasonable and natural. Assume λ smooth =0.1, the smoothing regularization term is calculated:

[0269]

[0270] S6. Define a reinforcement learning model to adaptively optimize the user psychological profile and update the user psychological profile in real time.

[0271] Specifically, define the state s in the reinforcement learning environment t and action a t Status t Including the user's current psychological portrait u profile and context information c t Action a t It refers to the optimization strategy selected by the model in its current state, such as adjusting data weights, updating model parameters, etc.

[0272] Furthermore, we design the reward function r t To evaluate action a t The reward function combines user feedback and system performance indicators, and the specific formula is:

[0273] r t =λ1·Feedback(s t ,a t )+λ2·Performance(s t ,a t )

[0274] Among them, Feedback(s t ,a t ) indicates the user's feedback on the current state and action, Performance(s t ,a t ) represents the system performance (such as accuracy, response time, etc.), λ1 and λ2 are the weight coefficients of feedback and performance, respectively.

[0275] Then use the Deep Q-Network (DQN) for policy optimization. DQN approximates the Q-value function Q(s,a) through a deep neural network and updates the policy to maximize the cumulative reward. The Q-value update formula is:

[0276]

[0277] Among them, α is the learning rate, γ is the discount factor, and s t+1 To perform action a t The next state after t For immediate rewards.

[0278] Furthermore, according to the current strategy, adaptive sampling is performed from multimodal data. Define the data sampling probability P(x i |s t ), according to the current state s t Dynamically adjust data sampling strategy:

[0279]

[0280] Among them, a i Indicates the selection of data x i action, Q(s t ,a i ) is the Q value of the action.

[0281] After executing the optimal strategy, update the user psychological portrait u profile . Define the update function f update , combined with the current state and action, adjust the user's psychological portrait in real time:

[0282] u profile (t+1)=f update (u profile (t),a t )

[0283] Among them, f update represents the updating mechanism of user psychological portrait, a t Indicates the current optimization action.

[0284] This step ensures the realization of reinforcement learning optimization and adaptive update. Define the state, action and reward function, optimize the strategy through DQN, realize adaptive data sampling and real-time update of user psychological portrait. The design of each step is closely connected and serves the overall goal of the patent. It solves the problems of inflexible model optimization and insufficient data utilization in the existing technology, and provides a solid foundation for the accuracy and dynamics of user psychological portrait.

[0285] The embodiment of the present invention further provides a user psychological portrait acquisition system for cross-modal data fusion, the system comprising:

[0286] A multimodal data acquisition unit, used to collect multimodal data of the user and pre-process the data; the multimodal data includes text data, image data, behavior data and social relationship data;

[0287] The vector embedding unit is used to embed the preprocessed multimodal data. The cross-modal alignment network is used to map the multimodal data to the same embedding space. The self-attention mechanism is introduced to perform weighted processing on the features in the embedding space. The weighted multimodal data is fused to obtain multimodal fusion features.

[0288] The emotion feature extraction unit is used to extract emotion features from the pre-processed multi-modal data, fuse the extracted emotion features with cross-modal emotion features, obtain a comprehensive emotion feature vector, and construct a time series emotion model to analyze the comprehensive emotion feature vector and capture the emotion change trend.

[0289] A user psychological portrait construction unit is used to construct a social relationship network based on the user's social relationship data, integrate the social relationship data with multimodal data, analyze the impact of social relationships on the user's psychological state, and obtain a user psychological portrait that integrates the social relationship data; wherein the social relationship data includes friend relationships, interaction frequency, and message exchange records;

[0290] A user psychological portrait correction unit is used to collect the user's situational information to design a scenario sensitivity score, judge the social environment according to the scenario sensitivity score, and correct the user psychological portrait based on the judgment result and the user psychological portrait of the social relationship data to obtain a final psychological portrait correction objective function; wherein the situational information includes geographic location, environmental noise level and device usage;

[0291] The user psychological portrait updating unit is used to define a reinforcement learning model to adaptively optimize the user psychological portrait and update the user psychological portrait in real time.

[0292] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0293] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0294] In the description of the present application, it should be noted that the terms "upper", "lower", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or the orientations or positional relationships in which the inventive product is usually placed when in use. These are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0295] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0296] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for obtaining user psychological portraits by cross-modal data fusion, characterized in that: The method comprises: S1. Collecting multimodal data of users and preprocessing the data; the multimodal data includes text data, image data, behavior data, social relationship data and audio data; S2. Embed the preprocessed multimodal data, use a cross-modal alignment network to map the multimodal data to the same embedding space, introduce a self-attention mechanism to embed the features in the space for weighted processing, fuse the weighted multimodal data, and obtain multimodal fusion features; S3, perform sentiment feature extraction on the multimodal fusion features, and perform cross-modal sentiment feature fusion on the extracted sentiment features to obtain a comprehensive sentiment feature vector, and construct a time series sentiment model to analyze the comprehensive sentiment feature vector to capture sentiment change trends; S4. Based on the user's social relationship data, a social relationship network is constructed to integrate the social relationship data with multimodal data, analyze the impact of social relationships on the user's psychological state, and obtain a user psychological portrait that integrates the social relationship data; wherein the social relationship data includes friend relationships, interaction frequency, and message exchange records; S5. Collect the user's situational information to design a scenario-sensitive score, judge the social environment based on the scenario-sensitive score, and correct the user's psychological profile based on the judgment result and the user's psychological profile integrated with the social relationship data to obtain the final psychological profile correction objective function; S6. According to the final psychological portrait correction objective function, a reinforcement learning model is defined to adaptively optimize the user psychological portrait, and the user psychological portrait is updated in real time; The correction of the user psychological portrait based on the judgment result and the integration of the user psychological portrait of the social relationship data includes a segmented correction mechanism and the introduction of a situational sensitivity weight, wherein the segmented correction mechanism includes linear adjustment and nonlinear adjustment; The scenario information includes geographic location, ambient noise level, and device usage; the social environment is judged based on the scenario sensitivity score, as shown below: S scene =f scene (c i )=σ(W s ·c i +b s ) Among them, f scene represents a deep neural network, including multiple hidden layers and an output layer; W s represents the weight matrix, b s represents the bias vector, σ represents the activation function, and the output S scene The value range of c is [0,1], where values ​​close to 1 represent sensitive scenarios and values ​​close to 0 represent non-sensitive scenarios. i Represents the scene feature vector, denoted as c i =[L i ; T i ; N i ;D i ], where L i The code representing the position, T i The code representing time, N i The code representing the noise level, D i A code indicating the usage of the device; The linear adjustment has a direct impact on the emotional dimension of scene sensitivity, and is expressed as follows: e adjusted,k (x i )=e fusion,k (x i )×(1-β k ·S scene ) Among them, e fusion,k (x i ) represents the original score of the kth sentiment dimension, S scene represents the scene sensitivity score, β k Represents the coefficient used to control the adjustment strength of this dimension; The nonlinear adjustment has a nonlinear dependence on the emotion dimension of scene sensitivity, and adopts exponential adjustment, which is expressed as follows: e adjusted,k (x i )=e fusion,k (x i )×exp(-γ k ·S scene ) Among them, γ k represents the adjustment coefficient, which determines the intensity of the response of this dimension to the scene sensitivity; The scenario sensitivity weight represents the sensitivity of each emotional dimension to the scenario, which is expressed as follows: And adjusted (x i )=and fusion (x i )×(1-ψ⊙S scene ) Among them, ψ represents the scenario sensitivity weight vector, which indicates the sensitivity of each emotional dimension; ⊙ is the Hadamard product, which is multiplied element by element. scene Rate the sensitivity of the scene.

2. The method for obtaining user psychological portraits by cross-modal data fusion according to claim 1, characterized in that: The preprocessing includes data cleaning, data standardization, and data synchronization and alignment; The data cleaning includes cleaning text data, denoising image data, denoising audio data, and deduplication, completion, and consistency checking of behavior data and social relationship data. The data standardization processing includes word vectorization of text data, normalization of image data, feature extraction of audio data and standardization of behavior data; The data synchronization and alignment synchronizes the multimodal data by timestamp, aligns them to the same time axis, and uses linear interpolation to complete the missing data.

3. The method for obtaining user psychological portraits by cross-modal data fusion according to claim 2, characterized in that: The use of linear interpolation to complete the missing data is expressed as follows: Among them, t1 and t2 represent adjacent time points, D sync (t) represents the interpolated data, D sync (t1) represents the data at t1, D sync (t2) represents the data at time t2.

4. The method for obtaining user psychological portraits by cross-modal data fusion according to claim 1, characterized in that: For text data, the BERT model is used for vector embedding and sentiment feature extraction; for image data, the ResNet50 model is used for vector embedding and sentiment feature extraction; for audio data, the VGGish model is used for vector embedding and sentiment feature extraction.

5. The method for obtaining user psychological portraits by cross-modal data fusion according to claim 4, characterized in that: Use a cross-modal alignment network f based on the features extracted from text data, image data, or audio data CMAN Mapping data of different modalities to the same embedding space is expressed as follows: v aligned (x i )=f CMAN (v t (x i ),v i (x i ),v a (x i )) Among them, x i represents text data, image data, or audio data, f CMAN represents the cross-modal alignment network, v aligned (x i ) represents the feature vector mapped to the same embedding space, v t 、v i and v a Represent the embedding vectors of text, image and audio respectively; The self-attention mechanism is used to weight the features in the embedding space, which can be expressed as follows: v att (x i )=W att ·v aligned (x i ) Among them, W att represents the self-attention weight matrix obtained through training, v att Represents the feature vector after processing by the self-attention mechanism; Adding regularization terms improves the generalization ability and robustness of the model, as shown below: Among them, R1(W) represents the regularization term of multimodal data, ∥W att ∥ F represents the Frobenius norm, v orig (x i ) represents the original feature vector, λ1 and λ2 are the weight coefficients of the regularization term respectively; Then the overall loss function L1 of multimodal data is expressed as follows: in, represents the alignment loss, which is used to measure the effect of feature alignment between different modalities; The weighted feature vectors of text, image, and audio processed by the self-attention mechanism are fused to generate a comprehensive multimodal feature representation, which is expressed as follows: v fusion (x i )=f fusion (v att (x i )) Among them, v fusion (x i ) represents the final fused multimodal feature vector, f fusion represents the fusion function; Using cross-modal sentiment fusion network f CMEFN The emotional features extracted from the weighted feature vectors of text, image and audio are fused into a comprehensive emotional feature vector e fusion (x i ), which is expressed as follows: e fusion (x i )=f CMEFN (e t (x i ),e i (x i ),e a (x i )) Among them, e t 、e i and e a Respectively represent the sentiment features extracted from the weighted feature vectors of text, image and audio.

6. The method for obtaining user psychological portraits by cross-modal data fusion according to claim 5, characterized in that: According to the comprehensive emotional feature vector e fusion (x i ) uses enhanced LSTM to build a time series emotion model and provide a dynamic emotional state representation, which is expressed as follows: s t =f E-LSTM (e fusion (x i:t ))+d Among them, e fusion (x i:t ) represents the sentiment feature vector in time series t, s t represents the emotional state vector of the current time step, δ is used to smooth emotional changes and prevent excessive emotional fluctuations, and f E-LSTM Represents the enhanced LSTM model function; Among them, in the process of emotional feature fusion and time series modeling, the emotional similarity regularization term R is added sim and the time series smoothing regularization term R smooth , the total regularization term R2(W) of the sentiment feature is obtained as: in, Used to maintain the similarity of the same user's emotional characteristics, Used to smooth the temporal emotional state, λ3 and λ4 are the weight coefficients of the total regularization term R2(W) of the emotional feature; The overall loss function L2 with emotional features is expressed as follows: in, Represents the loss of emotion recognition and temporal modeling, which is used to measure the effect of emotion feature extraction and temporal modeling.

7. The method for obtaining user psychological portraits by cross-modal data fusion according to claim 1, characterized in that: The S4 specifically includes: S401: construct a social relationship network based on social relationship data, wherein the social relationship network is represented as a graph structure in, represents the set of user nodes, ε represents the set of edges between users, and the weight of the edge is w ij It is determined by the interaction frequency and message exchange volume between user i and user j, which can be expressed as follows: w ij =α·f ij +b·m ij Among them, f ij represents the interaction frequency between user i and user j, m ij represents the amount of message exchanges, α and β are weight coefficients, controlling the impact of interaction frequency and message traffic flow; S402. Each user node Contains the user's multimodal emotional feature vector e fusion (x i ) and social relationship feature vector r i , forming the node feature vector h i ; S403: Use graph convolutional neural networks to extract features and propagate information on social networks. By superimposing multiple layers of graph convolutional neural networks, the aggregation and information propagation of social network node features are achieved, and finally a high-dimensional feature representation of each node is obtained. It is expressed as follows: H (l+1) =σ(AH (l) W (l) Among them, H (l) is the feature matrix of the l-th layer node, A is the normalized adjacency matrix, W (l) is the weight matrix of the lth layer, σ(·) is the activation function, H (l+1) is the node feature matrix of the l+1th layer; S404. Adding social relationship consistency regularization term to graph convolutional neural network and feature similarity regularization term The total regularization term is for: in, Used to maintain the consistency of social relationship nodes, Used to maintain feature similarity, λ5 and λ6 are the weight coefficients of the regularization term, represents the high-dimensional feature representation i, Represents high-dimensional feature representation j, h i represents low-dimensional feature representation; The overall loss function L GCN-total for: in, Represents the loss of graph convolutional neural network, which is used to measure the effect of feature extraction and information propagation; S405. Represent the high-dimensional features output by the graph convolutional neural network and the user's emotional state vector s t Fusion, generate the user's comprehensive portrait feature vector p i , expressed as: Among them, f fuse is a fusion function used to combine high-dimensional features and emotional states.

8. A user psychological portrait acquisition system based on cross-modal data fusion, characterized in that: The system comprises: A multimodal data acquisition unit, used to collect multimodal data of users and pre-process the data; the multimodal data includes text data, image data, behavior data and social relationship data; The vector embedding unit is used to embed the preprocessed multimodal data. The cross-modal alignment network is used to map the multimodal data to the same embedding space. The self-attention mechanism is introduced to perform weighted processing on the features in the embedding space. The weighted multimodal data is fused to obtain multimodal fusion features. The emotion feature extraction unit is used to extract emotion features from the pre-processed multi-modal data, fuse the extracted emotion features with cross-modal emotion features, obtain a comprehensive emotion feature vector, and construct a time series emotion model to analyze the comprehensive emotion feature vector and capture the emotion change trend. A user psychological portrait construction unit is used to construct a social relationship network based on the user's social relationship data, integrate the social relationship data with multimodal data, analyze the impact of social relationships on the user's psychological state, and obtain a user psychological portrait that integrates the social relationship data; wherein the social relationship data includes friend relationships, interaction frequency, and message exchange records; A user psychological portrait correction unit is used to collect the user's situational information to design a scenario sensitivity score, judge the social environment according to the scenario sensitivity score, and correct the user psychological portrait based on the judgment result and the user psychological portrait of the social relationship data to obtain a final psychological portrait correction objective function; wherein the situational information includes geographic location, environmental noise level and device usage; A user psychological profile update unit, which is used to define a reinforcement learning model to adaptively optimize the user psychological profile and update the user psychological profile in real time; The correction of the user psychological portrait based on the judgment result and the integration of the user psychological portrait of the social relationship data includes a segmented correction mechanism and the introduction of a situational sensitivity weight, wherein the segmented correction mechanism includes linear adjustment and nonlinear adjustment; The scenario information includes geographic location, ambient noise level, and device usage; the social environment is judged based on the scenario sensitivity score, as shown below: S scene =f scene (c i )=σ(W s ·c i +b s ) Among them, f scene represents a deep neural network, including multiple hidden layers and an output layer; W s represents the weight matrix, b s represents the bias vector, σ represents the activation function, and the output S scene The value range of c is [0,1], where values ​​close to 1 represent sensitive scenarios and values ​​close to 0 represent non-sensitive scenarios. i Represents the scene feature vector, denoted as c i =[L i ; T i ; N i ;D i ], where L i The code representing the position, T i The code representing time, N i The code representing the noise level, D i A code indicating the usage of the device; The linear adjustment has a direct impact on the emotional dimension of scene sensitivity, and is expressed as follows: e adjusted,k (x i )=e fusion,k (x i )×(1-β k ·S scene ) Among them, e fusion,k (x i ) represents the original score of the kth sentiment dimension, S scene represents the scene sensitivity score, β k Represents the coefficient used to control the adjustment strength of this dimension; The nonlinear adjustment has a nonlinear dependence on the emotion dimension of scene sensitivity, and adopts exponential adjustment, which is expressed as follows: e adjusted,k (x i )=e fusion,k (x i )×exp(-γ k ·S scene ) Among them, γ k represents the adjustment coefficient, which determines the intensity of the response of this dimension to the scene sensitivity; The scenario sensitivity weight represents the sensitivity of each emotional dimension to the scenario, which is expressed as follows: And adjusted (x i )=and fusion (x i )×(1-ψ⊙S scene ) Among them, ψ represents the scenario sensitivity weight vector, which indicates the sensitivity of each emotional dimension; ⊙ is the Hadamard product, which is multiplied element by element. scene Rate the sensitivity of the scene.

Images