User portrait construction method, system and equipment based on cross-modal alignment

Abstract

The invention discloses a user portrait construction method, system and device based on cross-modal alignment, and the method comprises the steps: carrying out the feature extraction of a plurality of modal data of the same user, and obtaining a plurality of feature vector data corresponding to the modal data; sequentially mapping each piece of feature vector data into the constructed shared semantic space to obtain an embedded vector matched with each piece of feature vector data; all the embedded vectors are processed through a cross-modal attention mechanism, processing results are fused to obtain cross-modal fusion data of each user, and the processing process comprises the steps that the weight of each embedded vector of each user is updated through a comparison loss function, and based on the entity relation extracted from the cross-modal fusion data, the cross-modal fusion data of each user are obtained; constructing a knowledge graph corresponding to the current user; and inputting the knowledge graph into the constructed user portrait generation model to obtain the user portrait of the current user. According to the method provided by the embodiment of the invention, the accurate user portrait can be constructed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, and device for constructing a user portrait based on cross-modal alignment. Background Art

[0002] In the field of e-commerce and personalized recommendations, the accuracy of user portraits directly affects the effectiveness of the recommendation system.

[0003] In existing technologies, single modal data is usually used to generate user tags. However, in the existing network ecosystem, different modal data exist. The user portraits generated by single modal data are rough and have high feature noise, which in turn affects the personalized recommendations and precision marketing of e-commerce platforms.

[0004] It can be seen that how to use different modal data to build accurate user portraits has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention

[0005] The present invention provides a method, system and device for constructing a user portrait based on cross-modal alignment to solve the problem that the user portrait currently generated using single-modal data is rough, so as to realize the construction of accurate user portraits using different modal data.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a method for constructing a user profile based on cross-modal alignment, the method comprising:

[0007] Performing feature extraction processing on multiple modal data of the same user to obtain multiple feature vector data corresponding to the modal data;

[0008] Mapping each of the feature vector data sequentially into the constructed shared semantic space to obtain an embedding vector matching each of the feature vector data;

[0009] Processing all of the embedding vectors using a cross-modal attention mechanism and fusing the processed results to obtain cross-modal fused data for each user, wherein the processing includes updating the weights of each embedding vector for each user using a contrastive loss function constructed from multiple modal data of the same user and same-modal data of different users;

[0010] Constructing a knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data;

[0011] The knowledge graph is input into the constructed user portrait generation model to obtain the user portrait of the current user.

[0012] As one preferred solution, the modal data includes at least text data, image data, voice data and behavior data;

[0013] The feature extraction process is performed on a plurality of modal data of the same user to obtain a plurality of feature vector data corresponding to the modal data, including:

[0014] Performing encoding processing on the text data to obtain a text hidden layer vector, and performing dimensionality reduction processing on the text hidden layer vector to obtain text feature vector data;

[0015] Performing feature embedding processing on the image data to obtain image feature vector data;

[0016] Performing bidirectional modeling processing on the speech data to obtain speech feature vector data;

[0017] Serializing the behavior data to obtain behavior feature vector data;

[0018] The text feature vector data, the image feature vector data, the voice feature vector data and the behavior feature vector data are integrated and processed to obtain the feature vector data.

[0019] As one preferred solution, mapping each of the feature vector data into the constructed shared semantic space in sequence to obtain an embedding vector matching each of the feature vector data includes:

[0020] Each of the feature vector data is input into the shared semantic space constructed based on adversarial training to obtain an embedding vector that matches the feature vector data.

[0021] As one of the preferred solutions, the process of constructing the contrast loss function includes:

[0022] Construct positive sample pairs based on multiple modal data of the same user, and construct negative sample pairs based on the same modal data of different users;

[0023] Performing similarity processing on the positive sample pair and the negative sample pair respectively to obtain the similarity between the positive sample pair and the similarity between the negative samples;

[0024] The similarity between the positive sample pairs and the similarity between the negative samples are processed using a temperature cross entropy algorithm to obtain the contrast loss function.

[0025] As one of the preferred solutions, the construction of the knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data includes:

[0026] Extracting entities and relationships from the cross-modal fusion data;

[0027] Triples are constructed based on the associations between the entities and the relationships, and a knowledge graph is constructed using several of the triples.

[0028] As one of the preferred solutions, inputting the knowledge graph into the constructed user portrait generation model to obtain the user portrait of the current user includes:

[0029] Use the gated recurrent sequence model to model the knowledge graph data within a preset time to obtain a short-term interest portrait;

[0030] Use the incremental graph neural network model to process knowledge graph data older than a preset time to obtain long-term interest portraits;

[0031] The short-term interest portrait and the long-term interest portrait are integrated to obtain a user portrait.

[0032] As one of the preferred solutions, the gated recurrent sequence model is used to perform data modeling on the knowledge graph data within a preset time to obtain a short-term interest portrait, including:

[0033] The user portrait is updated, and the updating process includes adjusting the weight of the short-term interest portrait through the timeliness weight to achieve optimization of the user portrait.

[0034] Another embodiment of the present invention provides a user profile construction system based on cross-modal alignment, including:

[0035] An extraction module, configured to perform feature extraction processing on a plurality of modal data of the same user, respectively, to obtain a plurality of feature vector data corresponding to the modal data;

[0036] A mapping module, configured to sequentially map each of the feature vector data into the constructed shared semantic space to obtain an embedding vector matching each of the feature vector data;

[0037] a processing module for processing all the embedding vectors using a cross-modal attention mechanism and fusing the processing results to obtain cross-modal fused data for each user, wherein the processing process includes updating the weights of each embedding vector for each user using a contrastive loss function constructed from multiple modal data of the same user and same-modal data of different users;

[0038] A construction module, configured to construct a knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data;

[0039] The generation module is used to input the knowledge graph into the constructed user portrait generation model to obtain the user portrait of the current user.

[0040] Another embodiment of the present invention provides a user portrait construction device based on cross-modal alignment, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the user portrait construction method based on cross-modal alignment as described above.

[0041] Yet another embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the user portrait construction method based on cross-modal alignment as described above is implemented.

[0042] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0043] The present invention extracts features from multiple modal data of the same user to obtain multiple feature vectors corresponding to the modal data. Each feature vector is sequentially mapped into a constructed shared semantic space to obtain an embedding vector that matches each feature vector. All embedding vectors are processed using a cross-modal attention mechanism, and the processed results are fused to obtain cross-modal fused data for each user. The processing includes updating the weights of each embedding vector for each user using a contrastive loss function constructed from multiple modal data of the same user and same-modal data of different users. A knowledge graph corresponding to the current user is constructed based on entity relationships extracted from the cross-modal fused data. The knowledge graph is input into a constructed user profile generation model to obtain a user profile for the current user. Compared with the prior art, the present invention processes data from different modalities using contrastive learning and a cross-modal attention mechanism, ensuring semantic consistency between features of different modalities in a shared space and constructing an accurate user profile. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 1 is a flow chart of a method for constructing a user profile based on cross-modal alignment in one embodiment of the present invention;

[0045] Figure 2 1 is a schematic structural diagram of a user portrait construction system based on cross-modal alignment in one embodiment of the present invention;

[0046] Figure 3 It is a structural diagram of a user portrait construction device based on cross-modal alignment in one embodiment of the present invention.

[0047] Reference numerals:

[0048] Among them, 11, extraction module; 12, mapping module; 13, processing module; 14, construction module; 15, generation module. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] In the description of this application, the terms "first," "second," "third," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first," "second," "third," etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0051] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the two components. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are for illustrative purposes only, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0052] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meanings as those commonly understood by those skilled in the art. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Those skilled in the art will understand the specific meanings of the above terms in this application in specific circumstances.

[0053] In the field of e-commerce and personalized recommendations, the accuracy of user portraits directly affects the effectiveness of the recommendation system.

[0054] In existing technologies, traditional user profile construction based on knowledge graphs primarily utilizes static labels generated from user behavior and structured data to capture relationships between entities. However, these methods are mostly unimodal or focus solely on user behavior data, failing to fully utilize multimodal information such as text, images, and voice. This inadequate use of modal information makes it difficult to comprehensively and accurately portray user interests and behavior patterns, resulting in inaccurate user profile construction.

[0055] In addition, traditional knowledge graphs rely on predefined static relationships, making it difficult to capture dynamic changes in users' interests. They also have weak cross-modal association modeling and are unable to establish fine-grained semantic correspondences, such as the association between the user comment "mountain climbing" and the outdoor scenes in the picture.

[0056] In order to build an accurate user portrait, an embodiment of the present invention provides a method for building a user portrait based on cross-modal alignment. For details, see Figure 1 , Figure 1 The figure shows a flow chart of a method for constructing a user profile based on cross-modal alignment in one embodiment of the present invention, the method comprising:

[0057] S1: performing feature extraction processing on multiple modal data of the same user to obtain multiple feature vector data corresponding to the modal data;

[0058] S2: Mapping each of the feature vector data into the constructed shared semantic space in sequence to obtain an embedding vector matching each of the feature vector data;

[0059] S3: Processing all the embedding vectors using a cross-modal attention mechanism and fusing the processing results to obtain cross-modal fused data for each user, wherein the processing process includes updating the weights of each embedding vector for each user using a contrastive loss function, wherein the contrastive loss function is constructed using multiple modal data of the same user and same-modal data of different users;

[0060] S4: constructing a knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data;

[0061] S5: Input the knowledge graph into the constructed user portrait generation model to obtain the user portrait of the current user.

[0062] Specifically, the multiple modal data include text data, image data, voice data and behavior data.

[0063] Text data includes user comments, search terms, product descriptions, and product attributes such as model, specifications, price, color, function, product type, and brand. During the data preprocessing phase, natural language processing techniques are used to segment and clean the text data, embedding it, and standardizing each data type.

[0064] The process of extracting features from text data includes encoding the text data to obtain a text hidden layer vector, and performing dimensionality reduction on the text hidden layer vector to obtain text feature vector data. Preferably, a pre-trained language model based on a Transformer structure (BERT model) is used to extract the semantic embedding feature vector of the text information, thereby accurately capturing the text semantics.

[0065] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture. It learns deep semantic representations of text through bidirectional context. The BERT model is pre-trained using two tasks: the "Masked Language Model" and the "Next Sentence Prediction" task, thereby learning rich semantic information.

[0066] The text vectorization process is as follows:

[0067] E text =BERT(Text)

[0068] Specifically, for the input text Text={w1,w2…,w n}, the BERT model first encodes it into word embedding to generate a high-dimensional sentence representation, and the output vector is represented as H = {h1,h2,…,h n}, where h i Represents the hidden layer vector of each word, and takes out the [CLS] vector h CLS As a sentence-level representation, it can be used to represent the semantic information of the entire sentence. The principal component analysis method (PCA) is used to reduce the dimensionality of the high-dimensional vector generated by BERT to obtain the feature vector E text .

[0069] Among them, PCA principal component analysis is a dimensionality reduction method used to convert high-dimensional data into low-dimensional representation while retaining the main information of the data as much as possible. It combines the original variables into a small number of principal components through linear transformation. These principal components are orthogonal to each other and are arranged in order according to the size of the variance.

[0070] The image data includes user review images, search images, product main images, and product detail images. The image data processing process includes feature embedding processing on the image data to obtain image feature vector data.

[0071] Preferably, in the data preprocessing stage, the product images are normalized to 224×224 by performing noise reduction, data enhancement and other operations, and the pre-trained ResNet-50 model is used to extract image feature embedding to obtain the feature vector E image .

[0072] E image =ResNet(image)

[0073] Among them, ResNet is a deep convolutional neural network model. Its core lies in the introduction of residual structure, which solves the degradation problem of deep networks and can efficiently extract the deep semantic features of images.

[0074] The speech data includes user speech comments and speech searches. The process of processing the speech data includes performing a bidirectional modeling process on the speech data to obtain speech feature vector data.

[0075] Preferably, the speech data is subjected to noise reduction processing in the data preprocessing stage, and features are extracted after frame windowing. The processed speech data is converted into Mel-frequency cepstral coefficient (MFCC) features to form the input of Bi-LSTM. The input features are bidirectionally modeled by bidirectional LSTM to obtain the front and back dependency information. The last layer output of Bi-LSTM is used as the semantic vector to obtain the feature vector E audio .

[0076] E audio =BiLSTM(audio)

[0077] The bidirectional long short-term memory (Bi-LSTM) network is a sequence processing model that extends the standard LSTM. By introducing memory cells, LSTM effectively captures long-term dependencies in sequence data, addressing the vanishing and exploding gradient problems of traditional RNN models. Bi-LSTM builds on LSTM by adding a bidirectional propagation mechanism, enabling the model to simultaneously capture forward and backward information in sequence data. For speech data, the feature vectors processed by Bi-LSTM can better represent the semantic information in speech.

[0078] Specifically, the processed speech data is converted into Mel-Frequency Cepstral Coefficient (MFCC) features, including Fourier transform and Mel-frequency scaling conversion of the speech data, to obtain an output MFCC feature matrix.

[0079] Behavior data includes user behavior sequences such as browsing, collecting, adding to cart, purchasing, etc. The processing of behavior data includes: serializing the behavior data to obtain behavior feature vector data.

[0080] Preferably, the Transformer model is used to encode the user behavior sequence to obtain the feature vector E behavior .

[0081] E behavior =Transformer(Behavior_Sequence)

[0082] The Transformer model, with its multi-head self-attention mechanism at its core, can capture interactive information between different modalities by calculating the correlation between elements in the input sequence. The text feature vector data, the image feature vector data, the speech feature vector data, and the behavior feature vector data are integrated and processed to obtain the feature vector data.

[0083] By embedding multimodal features such as text and images into a unified feature space, the relationship between users and products can be efficiently modeled. Therefore, each feature vector data is mapped to the constructed shared semantic space in turn to obtain an embedding vector that matches each feature vector data. Specifically, several feature vector data are input into the shared semantic space constructed based on the weight matrix and bias term to obtain an embedding vector that matches the feature vector data.

[0084] Specifically, design a mapping network f m (·), for each mode m, the initial features E of each mode m Mapped into a unified shared semantic space to obtain a new embedding vector z m .

[0085] z m =f m (E m )=W m E m +b m

[0086] Among them, m∈{text,image,audio,behavior},W m is the weight matrix of modality m, which is used to convert the feature vector of the mth modality into a scalar for calculating the attention score. It plays the role of learning the importance of the modality and determines the contribution of the modality feature in the shared semantic space. mThe bias term for the mth modality adjusts the mapped output so that the scores of different modalities can be compared on the same basis, compensating for inherent biases in the data. This way, the embedding vectors of different modalities are in the same space, allowing for direct comparison and similarity calculation.

[0087] Preferably, each of the feature vector data may be input into the shared semantic space constructed based on adversarial training to obtain an embedding vector matching the feature vector data.

[0088] In step S3, all the embedding vectors are processed using a cross-modal attention mechanism, and the processing results are fused to obtain cross-modal fusion data for each user. The processing process includes updating the weights of each embedding vector of each user using a contrastive loss function, and the contrastive loss function is constructed by multiple modal data of the same user and same-modal data of different users.

[0089] Specifically, the process of constructing the contrastive loss function first includes: constructing positive sample pairs based on several modal data of the same user, such as text-image, voice-behavior, and constructing negative sample pairs based on the same modal data of different users; performing similarity processing on the positive sample pairs and the negative sample pairs respectively to obtain the similarity of the positive sample pairs and the similarity between the negative samples; using the temperature cross entropy algorithm to process the similarity of the positive sample pairs and the similarity between the negative samples to obtain the contrastive loss function.

[0090] Contrastive loss function The goal is to reduce the distance between positive pairs in the shared semantic space and increase the distance between negative pairs.

[0091] Assume that for a positive sample pair (representation of the same product or user in different modalities) and a set of negative samples (modal representation of other products or users), Loss can be defined as:

[0092]

[0093] Where τ is the temperature hyperparameter (default 0.07) used to adjust the distribution smoothness, N is the number of negative samples, and sim(·) is calculated using cosine similarity:

[0094]

[0095] Specifically, the cross-modal attention mechanism is used to dynamically adjust the importance of features from different modalities to ensure semantic consistency in the shared space. The specific implementation steps are as follows.

[0096] For each modality m, calculate the attention weight:

[0097]

[0098] Embedding vector z of all modalities m Normalized to the probability distribution α m . So the weight α of each mode m Indicates the relative importance of the modality in fusion, and satisfies each α m is a positive number, α for all modes m The sum is 1. m The role of is to compare and normalize the scores of different modes, so that the mode with higher scores gets a larger weight and the mode with lower scores gets a smaller weight. m Calculate cross-modal fusion data z align .

[0099]

[0100] This vector ensures the consistency of different modal features in a shared semantic space, thereby achieving cross-modal alignment.

[0101] During the training process, the contrastive loss function and the cross-modal attention mechanism are jointly trained to obtain cross-modal fusion data. The attention mechanism dynamically adjusts the weights of different modal features through cross-modal interaction to narrow the distribution differences. The contrastive loss function enforces the similarity between modalities. The combination of the two can more efficiently learn cross-modal mapping.

[0102] Extract entity relationships from cross-modal fusion data, such as extracting entities such as users, products, brands, categories, and interest tags from user and product data. Extract relationships between users and entities such as products, brands, and categories from user behavior records. These relationships include browsing, favorites, and purchases between users and products, the affiliation of products with categories, and similar and complementary associations between products, such as "user-like-product" and "user-preference-brand."

[0103] Specifically, we first define the nodes and edges. The nodes include: user node: u∈U, which represents the user and his / her interests; product node: p∈P, which represents the product and its attributes; brand node: b∈B, which represents the brand and its category.

[0104] Edges include user-product edges: representing the interaction between users and products, such as browsing and purchasing; and product-brand edges: representing the affiliation between products and brands.

[0105] The extracted entities and relationships are constructed into a knowledge graph, represented in the form of triples, and stored in a graph database for subsequent fast query and update.

[0106]

[0107] Where ε is a node set (user node, product node, brand node), It is a set of edges (such as user-product edges, product-brand edges).

[0108] The structured representation corresponding to the knowledge graph is:

[0109] (User A) — [Browse] → (Product: Smartwatch);

[0110] (User A) — [Purchase] → (Product: Sports Headphones);

[0111] (Product: Smart Watch) — [Belongs to] → (Category: Smart Wearable Devices);

[0112] (Product: Smart Watch) — [Same Brand] → (Product: Smart Bracelet)

[0113] (User B)—[Purchase]→(Product: Smart Bracelet);

[0114] (User A)—[Similar interests]→(User B).

[0115] Mapping cross-modal fusion data to nodes in the knowledge graph includes:

[0116] h 用户 =Align(z text ,z image ,z audio ,z behavior )

[0117] h 商品 =Align(z text ,z image )

[0118] h 品牌 =Align(z text ,z image )

[0119] In the existing environment, user behavior and product data are constantly changing, and traditional static graph models cannot reflect the latest information in real time.

[0120] To this end, an incremental graph neural network model is used to process the knowledge graph. There is an incremental update mechanism in the incremental graph neural network model. When an entity that is not in the graph appears in the user behavior (such as the new brand "Y Brand"), an update is triggered and new nodes and new edges are added.

[0121]

[0122] in, is the feature of node i in layer l, N (i) is the neighbor set of node i, W l is the learnable weight matrix of layer l, and σ is the incremental update mechanism.

[0123] Preferably, the knowledge graph is input into the constructed user portrait generation model to obtain the user portrait of the current user, and a time range is preset, taking 48 hours as an example:

[0124] The gated recurrent sequence model is used to model the knowledge graph data within 48 hours to obtain short-term interest portraits. The incremental graph neural network model is used to process the knowledge graph data greater than 48 hours to obtain long-term interest portraits.

[0125] The short-term interest portrait and the long-term interest portrait are integrated to obtain a user portrait.

[0126] Specifically, short-term interests are extracted from users' recent behaviors, focusing on reflecting immediate needs:

[0127]

[0128] It is the feature vector of the product that the user interacted with most recently t times.

[0129] Long-term interests are based on the relationship between users and products in the knowledge graph. The IGNN model is used to mine high-order relationships in the graph, such as "User A-Preference-Smart Wearable Devices", to characterize the user's long-term preferences.

[0130] U long =IGNN(h u ,E N(u) )

[0131] h u is the embedding of the user in the knowledge graph, and N(u) is the set of neighbor nodes of the user (such as the products and tags that the user has interacted with).

[0132] By integrating short-term interests and long-term interests, we can obtain the initial user portrait, so as to accurately grasp the user's interest characteristics in different time spans.

[0133] U fusion =λU long +(1-λ)U short

[0134] Among them, λ controls the hyperparameter of the ratio of long-term to short-term interest, and the default value is 0.4. fusion Represents user portrait, U long Indicates long-term interest portrait, U short Indicates short-term interest profile.

[0135] Preferably, in the process of optimizing the user profile, a timeliness weight is assigned to the edge to reflect the change of interest, and recent behavior has a higher weight.

[0136] After obtaining the user portrait, it is necessary to optimize the portrait. Preferably, this embodiment optimizes the graph embedding loss and the user portrait loss, including:

[0137] The knowledge graph embedding loss aims to learn a low-dimensional vector representation of each entity and relationship, so that the triples (h, r, t) in the knowledge graph can be embedded into a unified vector space while preserving their semantic relationships. The knowledge graph embedding loss optimizes the geometric relationship of the knowledge graph triples through the formula, and the implementation method is as follows:

[0138] The triple form in the knowledge graph is (h, r, t), which represents the head entity h, the relation r, and the tail entity t. Positive samples are real triples (h, r, t), and negative samples are artificially constructed erroneous triples (h′, r, t′).

[0139] Negative samples are generated by negative sampling, and the model is ensured to distinguish between positive and negative samples. The graph embedding loss is:

[0140]

[0141] Among them, d() is the distance function (cosine distance), γ is the boundary threshold (default 0.1), h is the head entity, r is the relationship, t is the tail entity, E represents the original vector of the entity, E h ′ is the original vector of the negative sample head entity h, E t ′ is the original vector of the negative sample tail entity; E h +E r ,E t is the original vector representation of the positive sample, E h ′+E r ,E t ′ is the original vector representation of the negative sample.

[0142] User portrait losses include:

[0143]

[0144] h u is the user embedding vector predicted by the model, h true is the real user feature vector, which can be generated through supervised learning based on user labels (such as "sports enthusiasts"), and MSE calculates the mean squared error between the predicted embedding and the real embedding:

[0145]

[0146] By jointly optimizing the model, we build a comprehensive model to improve the generalization and overall performance of the user profile generation model. This is achieved as follows:

[0147]

[0148] λ1 is the weight of the graph embedding loss task, and λ2 is the weight of the user portrait loss task.

[0149] Another embodiment of the present invention provides a user portrait construction system based on cross-modal alignment. For details, see Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a user profile construction system based on cross-modal alignment in one embodiment of the present invention. The system includes:

[0150] An extraction module, configured to perform feature extraction processing on a plurality of modal data of the same user, respectively, to obtain a plurality of feature vector data corresponding to the modal data;

[0151] A mapping module, configured to sequentially map each of the feature vector data into the constructed shared semantic space to obtain an embedding vector matching each of the feature vector data;

[0152] a processing module for processing all the embedding vectors using a cross-modal attention mechanism and fusing the processing results to obtain cross-modal fused data for each user, wherein the processing process includes updating the weights of each embedding vector for each user using a contrastive loss function constructed from multiple modal data of the same user and same-modal data of different users;

[0153] A construction module, configured to construct a knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data;

[0154] The generation module is used to input the knowledge graph into the constructed user portrait generation model to obtain the user portrait of the current user.

[0155] See also Figure 3 , which is a structural block diagram of a user portrait construction device based on cross-modal alignment provided by an embodiment of the present invention. The user portrait construction device 20 based on cross-modal alignment provided by an embodiment of the present invention includes a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21. When the processor 21 executes the computer program, the steps in the embodiment of the user portrait construction method based on cross-modal alignment are implemented, for example Figure 1 or, when the processor 21 executes the computer program, the functions of the modules in the above-mentioned device embodiments, such as the extraction module 11, are implemented.

[0156] Exemplarily, the computer program can be divided into one or more modules, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the user profile construction device 20 based on cross-modal alignment. For example, the computer program can be divided into an extraction module 11, a mapping module 12, and a processing module 13. The specific functions of each module are as follows:

[0157] Extraction module 11, used to perform feature extraction processing on multiple modal data of the same user, and obtain multiple feature vector data corresponding to the modal data;

[0158] A mapping module 12 is configured to sequentially map each of the feature vector data into the constructed shared semantic space to obtain an embedding vector that matches each of the feature vector data;

[0159] A processing module 13 is configured to process all of the embedding vectors using a cross-modal attention mechanism and fuse the processing results to obtain cross-modal fused data for each user. The processing process includes updating the weights of each embedding vector for each user using a contrastive loss function constructed from multiple modal data of the same user and same-modal data of different users.

[0160] A construction module 14 is configured to construct a knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data;

[0161] The generation module 15 is used to input the knowledge graph into the constructed user portrait generation model to obtain the user portrait of the current user.

[0162] The user profile construction device 20 based on cross-modal alignment may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of a user profile construction device based on cross-modal alignment and does not limit the user profile construction device 20 based on cross-modal alignment. The device 20 may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the user profile construction device 20 based on cross-modal alignment may also include input and output devices, network access devices, buses, and the like.

[0163] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the user profile construction device 20 based on cross-modal alignment, and utilizes various interfaces and lines to connect the various parts of the user profile construction device 20 based on cross-modal alignment.

[0164] The memory 22 can be used to store the computer programs and / or modules, and the processor 21 implements the various functions of the user portrait construction device 20 based on cross-modal alignment by running or executing the computer programs and / or modules stored in the memory 22, and calling the data stored in the memory 22.

[0165] Among them, if the module integrated in the user portrait construction device 20 based on cross-modal alignment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0166] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0167] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to perform the steps in the method for constructing a user portrait based on cross-modal alignment as described in the above embodiment, for example Figure 1 Steps S1 to S5 described in .

[0168] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0169] The present invention extracts features from multiple modal data of the same user to obtain multiple feature vectors corresponding to the modal data. Each feature vector is sequentially mapped into a constructed shared semantic space to obtain an embedding vector that matches each feature vector. All embedding vectors are processed using a cross-modal attention mechanism, and the processed results are fused to obtain cross-modal fused data for each user. The processing includes updating the weights of each embedding vector for each user using a contrastive loss function constructed from multiple modal data of the same user and same-modal data of different users. A knowledge graph corresponding to the current user is constructed based on entity relationships extracted from the cross-modal fused data. The knowledge graph is input into a constructed user profile generation model to obtain a user profile for the current user. Compared with the prior art, the present invention processes data from different modalities using contrastive learning and a cross-modal attention mechanism, ensuring semantic consistency between features of different modalities in a shared space and constructing an accurate user profile.

[0170] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for constructing a user portrait based on cross-modal alignment, characterized in that: include: Performing feature extraction processing on multiple modal data of the same user to obtain multiple feature vector data corresponding to the modal data; Mapping each of the feature vector data sequentially into the constructed shared semantic space to obtain an embedding vector matching each of the feature vector data; Processing all of the embedding vectors using a cross-modal attention mechanism and fusing the processed results to obtain cross-modal fused data for each user, wherein the processing includes updating the weights of each embedding vector for each user using a contrastive loss function constructed from multiple modal data of the same user and same-modal data of different users; Constructing a knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data; The knowledge graph is input into the constructed user portrait generation model to obtain the user portrait of the current user.

2. The method for constructing a user profile based on cross-modal alignment according to claim 1, wherein: The modality data includes at least text data, image data, voice data and behavior data; The feature extraction process is performed on a plurality of modal data of the same user to obtain a plurality of feature vector data corresponding to the modal data, including: Performing encoding processing on the text data to obtain a text hidden layer vector, and performing dimensionality reduction processing on the text hidden layer vector to obtain text feature vector data; Performing feature embedding processing on the image data to obtain image feature vector data; Performing bidirectional modeling processing on the speech data to obtain speech feature vector data; Serializing the behavior data to obtain behavior feature vector data; The text feature vector data, the image feature vector data, the voice feature vector data and the behavior feature vector data are integrated and processed to obtain the feature vector data.

3. The method for constructing a user profile based on cross-modal alignment according to claim 1, wherein: Mapping each of the feature vector data sequentially to the constructed shared semantic space to obtain an embedding vector matching each of the feature vector data includes: Each of the feature vector data is input into the shared semantic space constructed based on adversarial training to obtain an embedding vector that matches the feature vector data.

4. The method for constructing a user profile based on cross-modal alignment according to claim 1, wherein: The construction process of the contrast loss function includes: Construct positive sample pairs based on multiple modal data of the same user, and construct negative sample pairs based on the same modal data of different users; Performing similarity processing on the positive sample pair and the negative sample pair respectively to obtain the similarity between the positive sample pair and the similarity between the negative samples; The similarity between the positive sample pairs and the similarity between the negative samples are processed using a temperature cross entropy algorithm to obtain the contrast loss function.

5. The method for constructing a user profile based on cross-modal alignment according to claim 1, wherein: The step of constructing a knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data includes: Extracting entities and relationships from the cross-modal fusion data; Triples are constructed based on the associations between the entities and the relationships, and a knowledge graph is constructed using several of the triples.

6. The method for constructing a user profile based on cross-modal alignment according to claim 1, wherein: Inputting the knowledge graph into the constructed user portrait generation model to obtain the user portrait of the current user includes: Use the gated recurrent sequence model to model the knowledge graph data within a preset time to obtain a short-term interest portrait; Use the incremental graph neural network model to process knowledge graph data older than a preset time to obtain long-term interest portraits; The short-term interest portrait and the long-term interest portrait are integrated to obtain a user portrait.

7. The method for constructing a user profile based on cross-modal alignment according to claim 6, wherein: The gated cyclic sequence model is used to model the knowledge graph data within a preset time to obtain a short-term interest portrait, including: The user portrait is updated, and the updating process includes adjusting the weight of the short-term interest portrait through the timeliness weight to achieve optimization of the user portrait.

8. A user portrait construction system based on cross-modal alignment, characterized by: include: An extraction module, configured to perform feature extraction processing on a plurality of modal data of the same user, respectively, to obtain a plurality of feature vector data corresponding to the modal data; A mapping module, configured to sequentially map each of the feature vector data into the constructed shared semantic space to obtain an embedding vector matching each of the feature vector data; a processing module for processing all the embedding vectors using a cross-modal attention mechanism and fusing the processing results to obtain cross-modal fused data for each user, wherein the processing process includes updating the weights of each embedding vector for each user using a contrastive loss function constructed from multiple modal data of the same user and same-modal data of different users; A construction module, configured to construct a knowledge graph corresponding to the current user based on the entity relationships extracted from the cross-modal fusion data; The generation module is used to input the knowledge graph into the constructed user portrait generation model to obtain the user portrait of the current user.

9. A user portrait construction device based on cross-modal alignment, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the user portrait construction method based on cross-modal alignment as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the user portrait construction method based on cross-modal alignment as described in any one of claims 1 to 7 is implemented.

Images