User profiling method, system, electronic device and medium

Abstract

The present disclosure relates to a user portrait method, system, electronic device and medium. The method includes: constructing a data set produced or consumed by user behavior into a data set to be analyzed including text and image data, and the image data includes N images; extracting features from the N images to obtain N global features; pooling the N global features and constructing N×M tokens; splicing the text data tokens with the N×M tokens and inputting them into a multimodal labeling model to obtain multimodal features; mapping the multimodal features to corresponding tags in a tag library through an FC layer; extracting features from the N images to obtain N global and face features; inputting global features, face features and text data into a data set analysis model to obtain keyword information; dividing the user's consumption behavior into three types of strong, weak and negative interactions and assigning different weights to them; and counting the corresponding tags and keyword information and weights of each data set produced or consumed by the user behavior over a period of time to obtain a user portrait.

Description

Technical Field

[0001] The present disclosure relates to the fields of machine learning and data processing, and more specifically to user profiling methods, systems, electronic devices, and media. Background Art

[0002] In social media platforms and online social media, users generate a large amount of information and data, including multimodal content such as text, images, and videos. Understanding and satisfying users' interests and preferences, and providing personalized information recommendations and search results are key goals of social media and platform platforms. However, this involves processing complex user behaviors and multimodal data, which requires efficient user understanding and interest modeling techniques.

[0003] At present, there is no complete user portrait construction solution for social scenarios in the market, especially the description accuracy of explicit portraits (labels) is required to be high. The current explicit portrait technology is often based on some words that can be qualitatively analyzed, such as gender, social identity, age, etc., but these labels cannot well describe a user's fine-grained preferences. First, in the social square, user interaction behaviors are complex and diverse, including posting, commenting, liking, sharing, etc. At the same time, the data generated by users includes text, images and other multimodal data, which makes it complicated to understand user behavior and preferences. Secondly, the existing user understanding technology mainly focuses on text analysis and simple recommendation systems, which is difficult to fully and accurately capture user interests and preferences. Traditional methods often rely on simple technologies such as keyword matching or collaborative filtering, which cannot handle multimodal data and complex user behaviors well. Finally, the existing technology usually only involves a single technical dimension when understanding user interests, and cannot provide the ability to fully explain user preferences. This leads to low recall and accuracy of interest content, and limited user personalized experience. Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] According to one aspect of the present disclosure, a user profiling method is provided, comprising: constructing each data set produced or consumed by user behavior into a data set to be analyzed including text data and image data, wherein the image data includes N images, wherein N≤4, and when the number of images is less than 4, the initialized image weights are added to keep the number of images spliced ​​with the text at 4; performing feature extraction on the N images in the data set to be analyzed respectively to obtain N global features; pooling the N global features and constructing N×M tokens, wherein M is the number of tokens for each image, and the larger the M, the more important the image data is in the training set; splicing the tokens of the text data with the N×M tokens, and inputting the spliced ​​tokens into a multimodal labeling model to obtain multimodal features; and inputting the multimodal features into a multimodal labeling model. The features are mapped to the corresponding labels in the label library through the fully connected layer; feature extraction is performed on the N images in the data set to be analyzed, and N global features and N facial features are obtained; the N global features, N facial features and the text data of the data set to be analyzed are input into the data set analysis model to obtain the keyword information after analysis; the user's consumption behavior is divided into three types: strong interaction, weak interaction and negative interaction, and the user's production behavior, strong interaction behavior and negative interaction behavior are respectively assigned a first weight, a second weight and a third weight, wherein the first weight is greater than the second weight and the third weight, and the first weight and the second weight are positive numbers, and the third weight is negative number; the corresponding labels and keyword information and their weights of each data set produced or consumed by the user behavior within a period of time are counted to obtain the user portrait.

[0006] According to some embodiments of the present disclosure, the modeling method of the data set analysis model includes: constructing a training data set including text data and image data, wherein the image data includes N images; performing feature extraction on the N images of the training data set respectively to obtain N global features and N facial features; adding a mask to the text data of the training data set to obtain masked text; inputting the N global features, N facial features and masked text into a multimodal pre-trained model to obtain predicted keyword information; inputting the predicted keyword information into a DeepCT model to obtain a predicted weight for each word in the predicted keyword information; and fine-tuning all parameters of the multimodal pre-trained model according to the difference between the predicted weight and the true weight of each word, and using the adjusted multimodal pre-trained model as the data set analysis model.

[0007] According to some embodiments of the present disclosure, adding a mask to the text data of the training data set to obtain the masked text includes: masking the keywords in the text data of the training data set to obtain the keyword-masked text.

[0008] According to some embodiments of the present disclosure, the modeling method of the multimodal labeling model includes: constructing a training data set including text data and image data, wherein the image data includes N images; performing feature extraction on the N images in the training data set respectively to obtain N global features; pooling the N global features and constructing N×M tokens; splicing the tokens of the text data with the N×M tokens, and inputting the spliced ​​tokens into the multimodal training model to obtain multimodal features; mapping the multimodal features to corresponding labels in the label library through a fully connected layer; adjusting the multimodal training model according to the difference between the mapped label and the true label, and using the adjusted multimodal training model as the multimodal labeling model.

[0009] According to some embodiments of the present disclosure, when the image data is video data, N images are obtained by cutting frames of the video data.

[0010] According to some embodiments of the present disclosure, when an image does not have local facial information, initialized weight information is used to replace the local facial information as facial features.

[0011] According to some embodiments of the present disclosure, the user portrait method also includes: constructing a real-time graph library; searching the real-time graph library through the obtained keyword information to obtain the search popularity of the relevant nodes corresponding to the keyword information; based on the search popularity of the relevant nodes corresponding to the keyword information, counting the corresponding labels and keyword information and their weights of each data set produced or consumed by the user behavior within a period of time to obtain a user portrait.

[0012] According to some embodiments of the present disclosure, building a real-time graph library includes: using the open source Ownthink graph database as a base library, deleting and / or expanding it for scenes specific to the site, and establishing a graph library; adding a heat field to each node based on the existing triples in the graph library, the field representing the degree of interest of users in the site in the node; and when new words appear, incorporating the new words into the existing triples in the graph library in real time to build a real-time graph library.

[0013] According to another aspect of the present disclosure, a user portrait system is provided, including a unit configured to execute the user portrait method of any embodiment described in the present disclosure.

[0014] According to some embodiments of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the user portrait method of any embodiment described in the present disclosure based on instructions stored in the memory.

[0015] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the user portrait method of any embodiment described in the present disclosure is executed.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the user portrait method of any embodiment described in the present disclosure is executed.

[0017] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings described herein are used to provide a further understanding of the present disclosure. Each of the drawings together with the following detailed description is included in this specification and forms a part of the specification, and is used to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation on the present disclosure. In the drawings:

[0019] Figure 1 An example flow chart of a user profiling method according to an embodiment of the present invention is shown.

[0020] Figure 2 An example flow chart of a method for training a multimodal labeling model according to an embodiment of the present invention is shown.

[0021] Figure 3 An example flow chart of a method for training a data set analysis model according to an embodiment of the present invention is shown.

[0022] Figure 4 An exemplary block diagram of a user portrait system according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0024] Figure 6 A block diagram showing an example structure of a computer system that may be employed in one embodiment according to the present disclosure.

[0025] It should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar parts. Therefore, once an item is defined in one drawing, it may not be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure, but it is obvious that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of the embodiments is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein.

[0027] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0028] The term "include" and its variations used in the present disclosure mean an open term including at least the following elements / features but not excluding other elements / features, i.e., "including but not limited to". In addition, the term "include" and its variations used in the present disclosure mean an open term including at least the following elements / features but not excluding other elements / features, i.e., "including but not limited to". Therefore, include is synonymous with include. The term "based on" means "based at least in part on".

[0029] References throughout this specification to "one embodiment," "some embodiments," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments." Moreover, the appearances of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment, but may do so.

[0030] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the concepts of "first", "second", etc. are not intended to imply that the objects described in this way must be in a given order in time, space, ranking, or any other manner.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined in any suitable manner that will be clear from the present disclosure by a person of ordinary skill in the art.

[0034] In order to solve the accuracy problem of understanding user interests and recommending content on social media platforms, this application adopts a multimodal data fusion method to classify data sets (for example, text and image information of posts on forums, video works on short video platforms, etc.), and applies multimodal keyword mining technology to capture important words and topics in the data set. This step aims to comprehensively consider the multimodal data generated by users, including text, images, etc., so as to more comprehensively understand the user's behavior and interests. Secondly, the present invention constructs an interest information model to comprehensively model the user's interests and preferences based on the user's production behavior and consumption behavior. The model not only takes into account the multimodal characteristics of user behavior, but also ranks the importance of user behavior to determine the content that users are more concerned about. The present invention constructs an interest graph with the help of a graph database structure. This graph database structure can be used to store user interest information, associate interest similarities between users, and reasonably expand the scope of user interests.

[0035] If you want to accurately label a user, you must first have the ability to accurately label the content. For the scenario of the data set, you can obtain up to 4 modal data, namely pictures, text, audio, and video. To this end, this application constructs a multimodal classification model for the data set scenario. Starting from the label system: The classification criteria of the model come from the points of interest of the user dimension, and a total of 450+ types of labels are designed. Starting from the data: different data mining methods are used, such as in-site search click data, image classification model data, and off-site crawler data, aiming to build a high-quality picture, text, and video data set. Starting from the model: The model structure is based on the user's production behavior (such as posting) habits, and supports the input of multiple data, such as plain text data, picture and text data, text and video data, etc. The method is to first use resnet152 and other methods to extract features from the image content. If there is no image, an initialization image is used as a placeholder. If it is video data, the video is cut to obtain the image. After passing through the image / video feature extraction network, the feature results of N images using resnet152 and other methods as feature extractors can be obtained; then this feature is pooled and mapped to a dimension such as 768, and M tokens are constructed for each image; the above image tokens and text tokens are concatenated, and the concatenated tokens are input into the multimodal training model to obtain the final multimodal features, and the final classification results of explicit labels are obtained after mapping through the fully connected layer.

[0036] Reference below Figures 1 to 4 The user profiling method and system according to an exemplary embodiment of the present invention will be described in detail.

[0037] According to an embodiment of the present invention, user profiling requires training a multimodal labeling model, and then labeling the data set produced or consumed by the user behavior based on the trained multimodal labeling model. According to an embodiment of the present invention, user profiling also requires training a word extraction model, i.e., a data set analysis model, and then performing keyword analysis on the data set produced or consumed by the user behavior based on the trained data set analysis model.

[0038] Figure 1 An example flowchart 100 of a user profiling method according to an embodiment of the present invention is shown.

[0039] like Figure 1 As shown, at step S101, for each data set produced or consumed by user behavior, a data set to be analyzed including text data and image data is constructed, wherein the image data includes N images.

[0040] The existing multimodal bi-directional transformer (MMBT: Multimodal Bitransformers) generally uses one image. However, posts on social forums often have multiple images. In order to improve the image learning ability, this application increases the image data to 4, and when the actual number of images is less than 4, the initialized image weights are added to keep the number of images spliced ​​with the text at 4.

[0041] Meanwhile, the image data in the data set of social forums or short video platforms are often video works. In this case, N images are obtained by cutting the video data. According to an exemplary embodiment of the present invention, the video data can be cut in a uniform way. For example, for a video with a duration of 40 seconds, when N is 4, the images at the 0th second, 10th second, 20th second, and 30th second are obtained by uniformly cutting the frames.

[0042] In step S102, feature extraction is performed on each of the N images in the data set to be analyzed to obtain N global features.

[0043] According to one embodiment of the present invention, resnet152 can be used to extract features from N images of the data set to be analyzed. According to an exemplary embodiment of the present invention, the dimension of the global feature includes but is not limited to 2048. It should be understood that obtaining the global features of the image based on resnet152 is only an example and should not be understood as a limitation on the global feature extraction method. The present invention can also use various other suitable global feature extraction methods, which will not be repeated here.

[0044] At step S103, the N global features are pooled and N×M tokens are constructed. It should be understood that M is the number of tokens for each image, and the larger M is, the more important the image data is in the data set.

[0045] According to an embodiment of the present invention, N global features are respectively pooled and mapped to a dimension of, for example, 768. It should be understood that the dimension 768 mapped here is only exemplary and may also be 512, 1024, etc.

[0046] In step S104, the token of the text data is concatenated with the N×M tokens, and the concatenated token is input into a multimodal labeling model to obtain a multimodal feature.

[0047] The multimodal labeling model here is a trained model. Figure 2 Describe in detail the training process of the multimodal labeling model. Figure 2 An example flowchart 200 of a method for training a multimodal labeling model according to an embodiment of the present invention is shown.

[0048] like Figure 2As shown, at step S201, a training data set including text data and image data is constructed, wherein the image data includes N images. According to an exemplary embodiment of the present invention, the number of images is increased to 4, and when the actual number of images is less than 4, the initialized image weights are added to keep the number of images spliced ​​with the text at 4. According to an embodiment of the present invention, the initialized weights can be represented by a set of all-0 vectors.

[0049] According to an exemplary embodiment of the present invention, when the image data is video data, N images are obtained by cutting frames of the video data.

[0050] According to an exemplary embodiment of the present invention, a training data set can be constructed by searching click data in the site, image classification model data, crawler data outside the site, etc., so as to construct a high-quality text, image and video training data set. It should be understood that the data in the site is the data of the social forum or platform to which the user portrait method of the present invention is applied.

[0051] In step S202, feature extraction is performed on N images in the training data set to obtain N global features.

[0052] According to an embodiment of the present invention, features of N images of the training data set may be extracted by using, for example, resnet152. According to an exemplary embodiment of the present invention, the dimension of the global feature includes, but is not limited to, 2048.

[0053] In step S203, the N global features are pooled and N×M tokens are constructed. M is the number of tokens for each image, and the larger the M, the more important the image data is in the data set.

[0054] In step S204, the tokens of the text data are concatenated with the N×M tokens, and the concatenated tokens are input into the multimodal training model to obtain multimodal features.

[0055] At step S205, the multimodal features are mapped to corresponding tags in the tag library through a fully connected layer. According to an embodiment of the present invention, the tag library has at least 450 types of tags, which are designed according to the interests of users in the station and can be updated in real time according to the changes in the interests of users.

[0056] In step S206, the multimodal training model is adjusted according to the difference between the mapped label and the true label, and the adjusted multimodal training model is used as the multimodal labeling model.

[0057] Back to Figure 1 , at step S105, the multimodal features are mapped to corresponding tags in the tag library through a fully connected layer.

[0058] In step S106, feature extraction is performed on each of the N images in the data set to be analyzed to obtain N global features and N facial features.

[0059] According to an embodiment of the present invention, extracting features from N images in the data set to be analyzed to obtain N global features includes: for each image, obtaining a global feature based on resnet101. According to an exemplary embodiment of the present invention, the dimension of the global feature includes but is not limited to 4096. It should be understood that obtaining the global features of an image based on resnet101 is only an example and should not be understood as a limitation on the global feature extraction method. The present invention can also adopt various other suitable global feature extraction methods, which are not described here.

[0060] According to one embodiment of the present invention, N images of the data set to be analyzed are subjected to feature extraction to obtain N facial features, including: for each image, extracting local facial information of the image based on Arcface as facial features. It should be understood that extracting local facial information of an image based on Arcface is only an example and should not be understood as a limitation on the method of extracting local facial information. The present invention may also adopt various other suitable local facial feature extraction methods, which are not described in detail here. According to an exemplary embodiment of the present invention, the dimensions of facial features include but are not limited to 512.

[0061] According to an exemplary embodiment of the present invention, when an image does not have local face information, initialized weight information is used instead of local face information as face features. It should be understood that the initialized weight information can indicate that the image is not a portrait. According to an embodiment of the present invention, the initialized weights can be represented by a set of all-0 vectors.

[0062] In step S107, N global features, N facial features and text data of the data set to be analyzed are input into a data set analysis model to obtain analyzed keyword information.

[0063] The dataset analysis model here is a trained model. Figure 3 Describe in detail the training process of the dataset analysis model. Figure 3 An example flowchart 300 of a method for training a data set analysis model according to an embodiment of the present invention is shown.

[0064] See also Figure 3 At step S301, a training data set including text data and image data is constructed, wherein the image data includes N images. According to an exemplary embodiment of the present invention, the number of images is increased to 4, and when the actual number of images is less than 4, the initialized image weights are added to keep the number of images spliced ​​with the text at 4.

[0065] According to an exemplary embodiment of the present invention, when the image data is video data, N images are obtained by cutting frames of the video data.

[0066] According to an exemplary embodiment of the present invention, a training data set can be constructed by searching and clicking data within the site. Using the site search and click data to create a training set can make the model more in line with the needs and interests of users within the site, better reflect the user's behavior and tendency, and further improve the effect of the model.

[0067] In step S302, feature extraction is performed on the N images of the training data set to obtain N global features and N facial features.

[0068] According to an embodiment of the present invention, extracting features from N images in a training data set to obtain N global features includes: for each image, obtaining a global feature based on, for example, resnet101. According to an exemplary embodiment of the present invention, the dimension of the global feature includes, but is not limited to, 4096.

[0069] According to an embodiment of the present invention, extracting features from N images of the data set to be analyzed to obtain N facial features includes: for each image, extracting local facial information of the image based on, for example, Arcface as facial features. According to an exemplary embodiment of the present invention, the dimensions of the facial features include, but are not limited to, 512.

[0070] According to an exemplary embodiment of the present invention, when an image does not have partial face information, initialized weight information is used to replace the partial face information as a face feature. It should be understood that the initialized weight information can indicate that the image is not a portrait image.

[0071] In step S303, a mask is added to the text data of the training data set to obtain masked text.

[0072] According to an embodiment of the present invention, adding a mask to the text data of the training data set to obtain the masked text includes: masking the keywords in the text data of the training data set to obtain the keyword-masked text.

[0073] In this field, traditional random masking will randomly extract a part (for example, about 15%) of tokens from a sentence for masking. This masking method can reflect the model's ability to understand the context in a sentence, but it cannot perform well in tasks with words as the smallest unit of measurement. Whole Word Masking Input will purposefully mask words composed of some characters, but this masking method often masks some less important words, causing deviations in the model during the learning process, thereby affecting the training effect. The method adopted in this application, Key Word Masking Input (KWM: Key Word Masking Input), is to mask the keywords in a sentence. The keyword masking input method can enhance the interaction between pictures and texts, and at the same time help the model understand the weight of the word importance model (DeepCT model) for keywords.

[0074] In step S304, N global features, N facial features, and masked text are input into a multimodal pre-training model to obtain predicted keyword information.

[0075] At step S305, the predicted keyword information is input into the DeepCT model to obtain the prediction weight of each word in the predicted keyword information.

[0076] In step S306, all parameters of the multimodal pre-trained model are fine-tuned according to the difference between the predicted weight and the true weight of each word, and the adjusted multimodal pre-trained model is used as a data set analysis model.

[0077] Deep Contextualized Term Weighting framework (DeepCT) is a way to provide term weights for search.

[0078] According to an exemplary embodiment of the present invention, the data clicked in the station is used to construct a training data set, and its core idea is to merge a series of post information clicked by users and complete the calculation of the probability of content appearance. For example, there are 8 posts in the post pool in the station: "Q1: ABC Concert", "Q2: ABC Ferris Wheel Live", "Q3: The Strongest Concert on the Surface", "Q4: ABC Top Ten Classic Images", "Q5: ABC Weibo Check-in", "Q6: ABC Concert Tickets", "Q7: All ABC Concerts", "Q8: Star Concert Live". According to the content of the posts in the post pool, the real weight of each word in the training data set "ABC Weibo data is so bad, it's hard to get a digit concert ticket" can be calculated. That is, "A", "B", "C" appear in a sentence with an importance of 6 / 8, "act", "sing", "meet" appear in a sentence with an importance of 5 / 8..., which is used as the real weight of each word.

[0079] For example, y is the predicted weight of each word in the predicted keyword information obtained by taking the post pool and the training dataset as input, y t is the true weight of each word in the training data set. Calculate the difference between the predicted weight and the true weight for each word.

[0080] All parameters of the multimodal pre-trained model are fine-tuned according to the calculated difference between the predicted weight and the true weight of each word, and the adjusted multimodal pre-trained model is used as a data set analysis model.

[0081] It should be understood that the smaller the difference between the predicted weight and the true weight of each word, the stronger the trained data set analysis model is, because the smaller the difference, the better the predicted keyword information fits the training data set.

[0082] Back to Figure 1 In step S108, the consumption behavior of the user is divided into three types: strong interaction, weak interaction and negative interaction, and the production behavior, strong interaction behavior and negative interaction behavior of the user are respectively assigned a first weight, a second weight and a third weight.

[0083] According to an exemplary embodiment of the present invention, user behaviors are divided into two categories: production behaviors and consumption behaviors. User consumption behaviors are further divided into strong interactive behaviors, weak interactive behaviors, and negative interactive behaviors. For example, strong interactive behaviors include but are not limited to likes, comments, favorites, and forwarding, weak interactive behaviors include but are not limited to clicks, and negative interactive behaviors include but are not limited to dislikes.

[0084] According to an exemplary embodiment of the present invention, each behavior of the user is assigned a different weight, among which production (for example, a user publishes a post or a video work) has the highest weight; strong interactive behaviors such as likes, comments, favorites, and forwarding have the second highest weight; clicking on a post is a weak interactive behavior and may not be weighted; disliking a post is a negative interactive behavior and is weighted down.

[0085] In step S109, the corresponding labels and keyword information and their weights of each data set produced or consumed by the user behavior within a period of time are counted to obtain a user portrait.

[0086] According to an embodiment of the present invention, each social square behavior of the user will be recorded by the user portrait model. For example, with 45 days as a cycle, the establishment of a long-term user portrait can be completed.

[0087] Although not shown in the figure, Figure 1 The user portrait method shown may also include constructing a real-time graph library, retrieving the real-time graph library through the obtained keyword information, obtaining the search popularity of the relevant nodes corresponding to the keyword information, and then based on the search popularity of the relevant nodes corresponding to the keyword information, counting the corresponding labels and keyword information and their weights of each data set produced or consumed by the user behavior over a period of time to obtain a user portrait.

[0088] The establishment of a real-time graph library can use an existing graph database (such as the open source Ownthink graph database with 200 million triples) as the base database, delete and / or expand it for the specific scenes in the site, and establish a graph library. Then, based on the existing triples in the graph library, add a heat field to each node of the triple, which represents the degree of interest of users in the site in the node, and when new words appear, merge the new words into the existing triples in the graph library in real time, so as to build a real-time graph library.

[0089] According to an embodiment of the present invention, in order to improve the usability of the graph, synonyms in the real-time graph library can be disambiguated and alias content can be merged. Using the networkX structural framework, millisecond-level graph database retrieval of 4.5 million entity words can be achieved.

[0090] The present invention can obtain coarse-grained labels (classification results) of the data set to be analyzed by applying the modal labeling model, and can obtain fine-grained labels (keyword extraction results) of the data set to be analyzed by applying the data set analysis model, thereby refining user interests and improving understanding of user interests. By implementing the establishment of a graph library, user interests can also be stored and the scope of user interests can be expanded, thereby enhancing generalization capabilities.

[0091] In this way, the present invention improves the recall rate and accuracy of user interest content, and can provide users with more targeted information recommendations. In addition, the present invention constructs an interest graph with the help of a graph database structure. This graph database structure can be used to store user interest information, associate interest similarities between users, and reasonably expand the scope of user interests. Through the interest graph, the user's long-term interest evolution and cross-domain interests can be better understood, providing more possibilities for personalized services. In short, the present invention achieves an all-round improvement in user interest understanding and content recommendation on social media platforms through key technologies such as multimodal data fusion, interest information modeling, and interest graph construction. This method not only improves the user experience, but is also expected to play an important role in multiple fields such as advertising, content recommendation, and search engine optimization. At the same time, the method has broad application prospects and can be applied to various multimodal data analysis and personalized recommendation scenarios.

[0092] Figure 4 An exemplary block diagram of a user portrait system 400 according to an embodiment of the present disclosure is shown. The user portrait system 400 according to the present invention can be implemented Figure 1 User profiling method 100 is shown.

[0093] like Figure 4 As shown, the user portrait system 400 includes a data set construction unit 41, which is configured to construct each data set produced or consumed by user behavior into a data set to be analyzed including text data and image data.

[0094] As referenced above Figure 1 As recorded, the image data includes N images, wherein when the image data is video data, N images are obtained by cutting frames of the video data, N is generally 4, and when the actual number of images is less than 4, initialized image weights are added to keep the number of images spliced ​​with the text at 4.

[0095] like Figure 4 As shown, the user portrait system 400 includes a first feature extraction unit 42, which is configured to perform feature extraction on N images in the data set to be analyzed respectively to obtain N global features.

[0096] According to an embodiment of the present invention, features of N images of the to-be-analyzed data set may be extracted by using, for example, resnet152. According to an exemplary embodiment of the present invention, the dimension of the global feature includes, but is not limited to, 2048.

[0097] like Figure 4 As shown, the user portrait system 400 includes a token construction unit 43 configured to pool N global features and construct N×M tokens. It should be understood that M is the number of tokens per image, and the larger M is, the more important the image data is in the training set.

[0098] like Figure 4 As shown, the user portrait system 400 includes a feature prediction unit 44, which is configured to concatenate the tokens of the text data with the N×M tokens, and input the concatenated tokens into a multimodal labeling model to obtain multimodal features.

[0099] like Figure 4 As shown, the user portrait system 400 includes a mapping unit 45, which is configured to map the multimodal features to corresponding tags in the tag library through a fully connected layer.

[0100] like Figure 4 As shown, the user portrait system 400 includes a second feature extraction unit 46, which is configured to perform feature extraction on N images in the data set to be analyzed, respectively, to obtain N global features and N facial features.

[0101] According to an embodiment of the present invention, extracting features from N images in the data set to be analyzed to obtain N global features includes: for each image, obtaining a global feature based on, for example, resnet101. According to an exemplary embodiment of the present invention, the dimension of the global feature includes, but is not limited to, 4096.

[0102] According to an embodiment of the present invention, extracting features from N images of the data set to be analyzed to obtain N facial features includes: for each image, extracting local facial information of the image based on, for example, Arcface as facial features. According to an exemplary embodiment of the present invention, the dimensions of the facial features include, but are not limited to, 512.

[0103] According to an exemplary embodiment of the present invention, when an image does not have partial face information, initialized weight information is used to replace the partial face information as a face feature. It should be understood that the initialized weight information can indicate that the image is not a portrait image.

[0104] like Figure 4 As shown, the user portrait system 400 includes a keyword prediction unit 47, which is configured to input N global features, N facial features and text data of the data set to be analyzed into a data set analysis model to obtain analyzed keyword information.

[0105] like Figure 4 As shown, the user portrait system 400 includes a weight adding unit 48, which is configured to divide the user's consumption behavior into three types: strong interaction, weak interaction and negative interaction, and assign a first weight, a second weight and a third weight to the user's production behavior, strong interaction behavior and negative interaction behavior, respectively.

[0106] According to an exemplary embodiment of the present invention, each behavior of the user is assigned a different weight, among which production (for example, a user publishes a post or a video work) has the highest weight; strong interactive behaviors such as likes, comments, favorites, and forwarding have the second highest weight; clicking on a post is a weak interactive behavior and may not be weighted; disliking a post is a negative interactive behavior and is weighted down.

[0107] like Figure 4 As shown, the user portrait system 400 includes a statistical unit 49, which is configured to count the corresponding tags and keyword information and their weights of each data set produced or consumed by user behavior within a period of time to obtain a user portrait.

[0108] According to an embodiment of the present invention, each social square behavior of the user will be recorded by the user portrait model. For example, with 45 days as a cycle, the establishment of a long-term user portrait can be completed.

[0109] Although not shown in the figure, the user portrait system 400 also includes a Figure 2 and Figure 3 The units of the various steps of the training method of the multimodal labeling model and the training method of the dataset analysis model shown in FIG.

[0110] Although not shown in the figure, the user portrait system 400 also includes a unit configured to build a real-time graph library; a unit that retrieves the real-time graph library by obtaining keyword information to obtain the search popularity of related nodes corresponding to the keyword information; and a unit that obtains the user portrait by counting the corresponding labels and keyword information and their weights of each data set produced or consumed by user behavior over a period of time based on the search popularity of related nodes corresponding to the keyword information.

[0111] The user portrait system provided by the embodiments of the present disclosure can implement the user portrait method provided by any embodiment in this document.

[0112] This application proposes a user profiling method and system based on multimodal data tagging, keyword extraction and interest graph construction, aiming to improve the efficiency of multimodal data analysis, achieve a full range of user preference interpretation, thereby improving the social media user experience and achieving more accurate interest understanding and content recommendation.

[0113] By applying the user portrait system and method proposed in this application, it is expected that the average number of interactions per person on the platform can be increased by 1.5%, the average browsing time for works can be increased by 1%, and the effective exposure of works can be increased by 1%.

[0114] Some embodiments of the present disclosure also provide an electronic device. Figure 5 The block diagram of some embodiments of the electronic device 5 of the present disclosure is shown. The electronic device can be used to implement the method according to any embodiment of the present disclosure.

[0115] For example, in some embodiments, the electronic device 5 can be various types of devices, such as, but not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. For example, the electronic device 5 can include a display panel for displaying data and / or execution results used in the scheme of the present disclosure. For example, the display panel can be in various shapes, such as a rectangular panel, an elliptical panel, or a polygonal panel. In addition, the display panel can be not only a flat panel, but also a curved panel, or even a spherical panel.

[0116] like Figure 5 As shown, the electronic device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51. It should be noted that Figure 5 The components of the electronic device 5 shown are only exemplary and non-restrictive. The electronic device 5 may also have other components according to actual application requirements. The processor 52 may control other components in the electronic device 5 to perform desired functions.

[0117] In some embodiments, the memory 51 is used to store one or more computer-readable instructions. When the processor 52 is used to execute the computer-readable instructions, the computer-readable instructions are executed by the processor 52 to implement the method according to any of the above embodiments. For the specific implementation of each step of the method and the related explanation content, please refer to the above embodiments, and the repeated parts will not be repeated here.

[0118] For example, the processor 52 and the memory 51 may communicate with each other directly or indirectly. For example, the processor 52 and the memory 51 may communicate with each other through a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 52 and the memory 51 may also communicate with each other through a system bus, which is not limited by the present disclosure.

[0119] For example, the processor 52 may be embodied as various appropriate processors, processing devices, etc., such as a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X86 or ARM architecture, etc. For example, the memory 51 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The memory 51 may include, for example, a system memory, which may store, for example, an operating system, an application, a boot loader, a database, and other programs. Various applications and various data may also be stored in the storage medium.

[0120] In addition, according to some embodiments of the present disclosure, various operations / processes according to the present disclosure, when implemented by software and / or firmware, can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 6 The computer system 600 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 6 A block diagram showing an example structure of a computer system that may be employed in an embodiment according to the present disclosure.

[0121] exist Figure 6 In the embodiment, the central processing unit (CPU) 601 performs various processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. In the RAM 603, data required when the CPU 601 performs various processes, etc. are also stored as needed. The central processing unit is only exemplary, and it can also be other types of processors, such as the various processors described above. The ROM 602, RAM 603 and the storage part 608 can be various forms of computer-readable storage media, as described below. It should be noted that although Figure 6 ROM 602, RAM 603 and storage section 608 are shown separately in FIG. 6 , but one or more of them may be combined or located in the same or different memory or storage modules.

[0122] The CPU 601, the ROM 602, and the RAM 603 are connected to one another via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0123] The following components are connected to the input / output interface 605: an input portion 606, such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output portion 607, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage portion 608, including a hard disk, a magnetic tape, etc.; and a communication portion 609, including a network interface card such as a LAN card, a modem, etc. The communication portion 609 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Figure 6 The various devices or modules in the computer system 600 shown in the figure communicate via the bus 604, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.

[0124] A drive 610 is also connected to the input / output interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0125] In the case where the above-described series of processing is implemented by software, a program constituting the software may be installed from a network such as the Internet or a storage medium such as the removable medium 611 .

[0126] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, or installed from the storage part 608, or installed from the ROM 602. When the computer program is executed by the CPU 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0127] It should be noted that in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0128] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0129] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute any of the methods of the above embodiments. For example, the instructions may be embodied as computer program codes.

[0130] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0133] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0134] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0135] In the description provided herein, many specific details are set forth. However, it is understood that embodiments of the present disclosure may be implemented without these specific details. In other cases, well-known methods, structures, and techniques are not presented in detail in order not to obscure the understanding of the description.

[0136] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0137] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A user profiling method, comprising: Each data set produced or consumed by user behavior is constructed as a data set to be analyzed including text data and image data, wherein the image data includes N images, where N≤4, and when the number of images is less than 4, the initialized image weights are added to keep the number of images spliced ​​with the text at 4; Perform feature extraction on each of the N images in the data set to be analyzed to obtain N global features; Pool the N global features and construct N×M tokens, where M is the number of tokens per image. The larger the M, the more important the image data is in the training set. Concatenate the token of the text data with the N×M tokens, and input the concatenated tokens into the multimodal labeling model to obtain multimodal features; Map the multimodal features to the corresponding labels in the label library through the fully connected layer; Perform feature extraction on N images in the data set to be analyzed, and obtain N global features and N face features; Input N global features, N facial features and text data of the data set to be analyzed into the data set analysis model to obtain keyword information after analysis; The consumption behaviors of users are divided into three types: strong interaction, weak interaction and negative interaction, and the production behaviors, strong interaction behaviors and negative interaction behaviors of users are respectively assigned the first weight, the second weight and the third weight, wherein the first weight is greater than the second weight and the third weight, and the first weight and the second weight are positive numbers, and the third weight is negative number; The corresponding labels, keyword information and weights of each data set produced or consumed by user behavior over a period of time are counted to obtain a user portrait.

2. The user profiling method according to claim 1, wherein the modeling method of the data set analysis model comprises: Construct a training data set including text data and image data, wherein the image data includes N images; Perform feature extraction on N images of the training data set to obtain N global features and N facial features; Add a mask to the text data of the training data set to obtain masked text; Input N global features, N facial features, and masked text into the multimodal pre-training model to obtain predicted keyword information; Input the predicted keyword information into the DeepCT model to obtain the prediction weight of each word in the predicted keyword information; as well as All parameters of the multimodal pre-trained model are fine-tuned according to the difference between the predicted weight and the true weight of each word, and the adjusted multimodal pre-trained model is used as a data set analysis model.

3. The user profiling method according to claim 2, wherein adding a mask to the text data of the training data set to obtain the masked text comprises: Mask the keywords in the text data of the training data set to obtain text with the keywords masked.

4. The user profiling method according to claim 1, wherein the modeling method of the multimodal labeling model comprises: Construct a training data set including text data and image data, wherein the image data includes N images; Perform feature extraction on N images in the training data set to obtain N global features; Pool N global features and construct N×M tokens; Concatenate the tokens of the text data with the N×M tokens, and input the concatenated tokens into the multimodal training model to obtain multimodal features; Map the multimodal features to the corresponding labels in the label library through the fully connected layer; The multimodal training model is adjusted according to the difference between the mapped label and the true label, and the adjusted multimodal training model is used as the multimodal labeling model.

5. A user portrait method as described in any one of claims 1-4, wherein when the image data is video data, N images are obtained by cutting frames of the video data.

6. A user portrait method as described in claim 1 or 2, wherein when the image does not have local facial information, initialized weight information is used to replace the local facial information as facial features.

7. The user profiling method according to claim 1, further comprising: Build a real-time atlas library; Through the obtained keyword information, the real-time graph library is searched to obtain the search popularity of the relevant nodes corresponding to the keyword information; According to the search popularity of the relevant nodes corresponding to the keyword information, the corresponding labels and keyword information and their weights of each data set produced or consumed by user behavior over a period of time are counted to obtain the user portrait.

8. The user profiling method according to claim 7, wherein building a real-time graph library comprises: Using the open source Ownthink graph database as the base database, we deleted and / or expanded it for the specific scenarios in the site to build a graph library; Based on the existing triples in the graph library, a heat field is added to each node, which represents the degree of interest of users on the site in the node; as well as When a new word appears, it is merged into the existing triples in the atlas library in real time to build a real-time atlas library.

9. A user profiling system, comprising a unit configured to execute the method as claimed in any one of claims 1 to 8.

10. An electronic device, comprising: Memory; and A processor coupled to the memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

Images