Social platform account portrait multi-modal vectorization representation method

Abstract

The invention relates to the technical field of internet big data, and discloses a social platform account portrait multi-modal vectorization characterization method, which comprises work content portrait characterization, account data portrait characterization and social relation portrait characterization. According to the multi-modal vectorization representation method for the account portrait of the social platform, the account portrait is modeled based on information such as the work content, the account data and the account social relation published by the user, the account feature is vectorized by using the multi-modal content, and the account information is represented as multi-dimensionally and comprehensively as possible, so that the account portrait is accurately constructed, and the user experience is improved. Through a feature vectorization technology, the problem of loss of text tag extraction information and the defect of manual design of text tags are solved, and meanwhile, through multiple dimensions such as work content, account data and account social relations and multi-modal content recognition and analysis, the problem that account portraits are not comprehensively represented only based on text information is solved.

Description

Technical Field

[0001] The present invention relates to the field of Internet big data technology, and specifically to a multimodal vectorized representation method for social platform account portraits. Background Art

[0002] The rapid development of social media and short video platforms has brought about an explosive growth in the number of users and content output. The behavior of platform accounts in the virtual world is a reflection of real people in the real world. The personal account information and content posted by users on these platforms not only reflect the users' values, interests, behavioral characteristics and social networks, but also contain a wealth of high-value information. In order to mine valuable information from these massive data, it is crucial to effectively analyze user accounts and work content. At present, through account profiling technology, it is possible to accurately model accounts on the platform, which provides solid technical support for applications such as user behavior analysis, personalized recommendations, and security monitoring.

[0003] The current account profiling technology is mainly based on extracting account-related text information and generating account profiling by labeling accounts. The relationship analysis of account profiling is also achieved by comparing the correlation between labels. However, the account profiling modeling technology based on text labeling has the following defects:

[0004] 1. There is a problem of information loss in the text label extraction process;

[0005] 2. Text labels rely on manual design. Whether the labels are comprehensive and accurate directly affects the quality of the account portrait.

[0006] 3. In the multimedia era, it is impossible to fully characterize account portraits based only on text information, and multimodal information is not taken into account. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a multimodal vectorized representation method for social platform account portraits, which models account portraits based on information such as the content of works published by users, account information, and account social relationships, and uses multimodal content to vectorize account features, so as to represent account information as multi-dimensionally and comprehensively as possible, thereby accurately constructing account portraits. The feature vectorization technology is used to solve the problems of information loss in text label extraction and defects in manually designed text labels. At the same time, through multiple dimensions such as work content, account information, account social relationships, and multimodal content recognition and analysis, the problem of incomplete account portrait representation based only on text information is solved.

[0008] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: 1. It includes work content portrait representation, account information portrait representation, social relationship portrait representation and account portrait representation, which is characterized in that the specific representation steps are as follows:

[0009] 1.1 Characterization of the content of the work:

[0010] Step 1: Obtaining work information;

[0011] Step 2: Data preprocessing;

[0012] Visual text extraction;

[0013] Speech to text extraction;

[0014] Video frame extraction;

[0015] Collection of works texts;

[0016] Collection of images of works;

[0017] Step 3: Multimodal feature extraction;

[0018] Step 4: Vectorize the content of the work;

[0019] Step 5: Characterization of the content of the work;

[0020] 1.2 Account Profile Representation:

[0021] Step 1: Obtain account information;

[0022] Step 2: Obtaining work information;

[0023] Step 3: Data preprocessing;

[0024] Visual text extraction;

[0025] Speech to text extraction;

[0026] Video frame extraction;

[0027] Collection of works texts;

[0028] Collection of images of works;

[0029] Step 4: Historical portrait feature extraction;

[0030] Step 5: Personal data feature extraction;

[0031] Step 6: Ethnic feature extraction

[0032] Semantic features of avatars;

[0033] Visual semantic features of the work;

[0034] Semantic features of the work’s text;

[0035] Step 7: Extract IP location change features;

[0036] Step 8: Account profile representation;

[0037] 1.3 Social relationship portrait representation:

[0038] Step 1: Obtain account relationship;

[0039] Step 2: Obtain account events;

[0040] Step 3: Knowledge graph construction;

[0041] Step 4: Extract social relationship vector;

[0042] Step 5: Social relationship portrait representation;

[0043] 1.4 Account portrait representation:

[0044] The account portrait is formed by the work content portrait, account information portrait and social relationship portrait, realizing the vectorization of the account portrait.

[0045] Furthermore, the sampling strategy in the data preprocessing of step 2 for characterizing the content portrait of the work is not limited to key frames, sampling frames at fixed time intervals, and extracting content representative frames through a frame sampling algorithm.

[0046] Furthermore, applicable types of cross-modal visual semantic features in the multimodal feature extraction of step three of the work content portrait representation include models with cross-modal visual semantic feature extraction capabilities.

[0047] Furthermore, in step 4 of representing the work content portrait, the aggregation methods in the work content portrait vectorization include but are not limited to averaging and fusion through a deep learning model.

[0048] Furthermore, the sampling strategy in the data preprocessing of step three of the account profile characterization is not limited to key frames, sampling frames at fixed time intervals, and extracting content representative frames through a frame sampling algorithm.

[0049] Furthermore, applicable types of cross-modal text semantic features in the personal data feature extraction in step five of the account profile representation include models with cross-modal text semantic feature extraction capabilities.

[0050] Furthermore, the applicable types of cross-modal text semantic features in the ethnic feature extraction of step six of the account profile characterization include models with the ability to extract cross-modal text semantic features, and a total of three features, namely, avatar semantic features, visual semantic features and text semantic features, are used as the ethnic features of the account.

[0051] Furthermore, the graph embedding algorithm in the social relationship vector extraction in step 4 of the social relationship portrait representation refers to an algorithm model with graph embedding capabilities, not limited to Node2Vec and DeepWalk.

[0052] Compared with the prior art, the present invention provides a multimodal vectorized representation method for social platform account portraits, which has the following beneficial effects:

[0053] 1. The multimodal vectorized representation method of social platform account portraits avoids information loss to the greatest extent by representing account portraits based on feature vectors.

[0054] 2. This multimodal vectorized representation method for social platform account portraits does not require manually designed text labels, has low usage costs, is widely applicable, and is highly universal.

[0055] 3. The multimodal vectorized representation method of the social platform account portrait comprehensively represents the account portrait from different dimensions, including the work content portrait, account information portrait and social relationship portrait, making the account portrait description more accurate and the application effect better. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flowchart of a multimodal vectorized representation method for social platform account portraits proposed by the present invention.

[0057] Figure 2 A flowchart for representing the content portrait of the work proposed by the present invention.

[0058] Figure 3 A flow chart of the account profile representation proposed by the present invention.

[0059] Figure 4 A flowchart of the social relationship portrait representation proposed by the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] See also Figure 1-4 A multimodal vectorized representation method for social platform account portraits includes work content portrait representation, account information portrait representation, social relationship portrait representation, and account portrait representation. The specific representation steps are as follows:

[0062] 1.1 Characterization of the content of the work:

[0063] By extracting the feature vector of the work content, the work content portrait representation of the account is realized. Specifically, the visual cross-modal semantic features are extracted from the cover image and video of the work, and the text cross-modal semantic features are extracted from the text content such as the work title, OCR, ASR, etc., and the visual and text semantic features are fused to form the portrait features of the account's works. It should be noted that OCR is optical character recognition and ASR is automatic speech recognition.

[0064] Step 1: Obtaining work information,

[0065] Obtain the account's work information, including published articles, pictures, video cover images, videos, and video titles;

[0066] Step 2: Data preprocessing,

[0067] Visual text extraction: Extract visual text that appears in images, video cover images, and videos through optical character recognition algorithms;

[0068] Speech-to-text extraction: Extract speech from the video and convert it into text through automatic speech recognition algorithm;

[0069] Video frame extraction: extract video frames using a certain sampling strategy. The sampling strategy is not limited to key frames, fixed time interval frame extraction, and content representative frames extracted through frame extraction algorithms.

[0070] Works text collection: Combine article text, visual text, and voice text into a works text collection;

[0071] Work image collection: merge the pictures, cover pictures and video frames in the work into an image collection;

[0072] Step 3: Multimodal feature extraction,

[0073] Extract cross-modal visual semantic features of an image set. One feature is extracted from one image. The cross-modal visual semantic features are not limited to a specific model structure. All existing models with the ability to extract cross-modal visual semantic features are applicable. Those skilled in the art can choose according to actual needs, and this solution does not limit them.

[0074] Extract cross-modal text semantic features of the work text, extract one feature for each text, cross-modal text semantic features are not limited to a specific model structure, existing models with cross-modal text semantic feature extraction capabilities are applicable, and those skilled in the art can choose according to actual needs, and this solution is not limited;

[0075] Step 4: Vectorize the work content portrait, aggregate all visual semantic features and text semantic features to form a visual semantic feature and a text semantic feature. The aggregation method includes but is not limited to averaging and fusion through deep learning models.

[0076] Step 5: Characterization of the content of the work.

[0077] The visual semantic features and textual semantic features generated in the feature aggregation step are used as the content portrait representation of the work.

[0078] 1.2 Account Profile Representation:

[0079] By comparing the similarity of account avatar features, personal information, ethnic characteristics, and IP location change patterns, the account profile representation is achieved. Among them, the ethnic characteristics of the account can be extracted by combining the visual and text information of the work;

[0080] Step 1: Obtain account information.

[0081] Obtain account information, including historical profile pictures, personal profiles, and historical IP locations;

[0082] Step 2: Obtaining work information,

[0083] Obtain the account's work information, including published articles, pictures, video cover images, videos, and video titles;

[0084] Step 3: Data preprocessing,

[0085] Visual text extraction: Extract visual text that appears in images, video cover images, and videos through optical character recognition algorithms;

[0086] Speech-to-text extraction: Extract speech from the video and convert it into text through automatic speech recognition algorithm;

[0087] Video frame extraction: Extract video frames using a certain sampling strategy. The sampling strategy is not limited to key frames, fixed time interval frame extraction, and content representative frames extracted through frame extraction algorithms.

[0088] Works text collection: Combine article text, visual text, and voice text into a works text collection;

[0089] Work image collection: merge the pictures, cover pictures and video frames in the work into an image collection;

[0090] Step 4: Historical avatar feature extraction,

[0091] Extract the copy detection features of historical avatars, one feature for each avatar;

[0092] Step 5: Personal data feature extraction,

[0093] Extracting semantic features of cross-modal text of personal profile. The semantic features of cross-modal text are not limited to a specific model structure. All existing models with the ability to extract semantic features of cross-modal text are applicable. Those skilled in the art can select them according to actual needs on the premise that they can extract semantic features of cross-modal text of personal profile.

[0094] Step 6: Extraction of ethnic characteristics,

[0095] Avatar semantic features: extract cross-modal visual semantic features of the latest avatars;

[0096] Visual semantic features of works: extract cross-modal visual semantic features of the above-mentioned collection of work images, and aggregate them to form a visual semantic feature. The cross-modal visual semantic feature is not limited to a specific model structure. Existing models with the ability to extract cross-modal visual semantic features are applicable. Those skilled in the art can select them according to actual needs on the premise that they can extract cross-modal visual semantic features of the collection of work images;

[0097] Work text semantic features: extract cross-modal text semantic features of the above-mentioned work text set, and aggregate them to form a text semantic feature. The cross-modal text semantic feature is not limited to a specific model structure. Existing models with the ability to extract cross-modal text semantic features are applicable. Those skilled in the art can select them according to actual needs on the premise that they can extract cross-modal text semantic features of the work text set;

[0098] A total of three features, namely, avatar semantic features, visual semantic features, and text semantic features, are used as the ethnic characteristics of the account;

[0099] Step 7: Extract IP location change features.

[0100] The historical IP location characteristics are extracted through the time series characterization model, which reflects the IP changes; the time series characterization model refers to an algorithm model with the ability to represent sequence characteristics. This embodiment does not limit a specific network structure, and those skilled in the art can select it according to actual needs, which is hereby explained;

[0101] Step 8: Account profile representation,

[0102] The four features of historical profile picture, personal data, ethnicity and IP location change are used together as account profile features to realize the vectorization of account profile.

[0103] 1.3 Social relationship portrait representation:

[0104] By constructing a social relationship graph between accounts and extracting account node vectors, we can achieve account social relationship vectorization and characterize social relationship portraits.

[0105] Step 1: Obtain account relationship.

[0106] Obtain information such as the account's followed accounts, fan accounts, comment information (including the commented account), and cross-platform association information. Cross-platform association information refers to social or short video platforms in the same system. Users can associate accounts on different platforms with each other to establish cross-platform association relationships.

[0107] Step 2: Obtain account events.

[0108] Get the topics that the account participates in as the account's events, and establish event association relationships between different accounts participating in the same event;

[0109] Step 3: Knowledge graph construction,

[0110] Build a relationship graph based on account relationships and event relationships. The nodes in the graph represent accounts or events, the edges represent the relationships between accounts, and the weights of the edges represent the degree of account interaction.

[0111] Relationships include: following, commenting, cross-platform synchronization, participating in topics, and having the same avatar;

[0112] The calculation method of the interaction degree of the following relationship is: if following, it is 1;

[0113] Method for calculating the interactive degree of comment relations: number of comments;

[0114] The calculation method of the interaction degree of the cross-platform synchronization relationship is: if it is a cross-platform association, it is 1;

[0115] The calculation method of the interaction degree of the topic relationship is: if participating in the same topic, it is 1;

[0116] The calculation method of the interaction degree of the same avatar relationship is: if the avatar is the same, it is 1;

[0117] Step 4: Extract social relationship vectors.

[0118] Use graph embedding algorithms to embed nodes (accounts) in social graphs into vector space and extract social relationship vectors. Graph embedding algorithms refer to algorithm models with graph embedding capabilities, not limited to Node2Vec and DeepWalk.

[0119] Step 5: Social relationship portrait representation,

[0120] Social relationship feature vector, as a representation of social relationship portrait.

[0121] At this point, the above-disclosed work content portrait representation, account information portrait representation, and social relationship portrait representation jointly construct the account portrait library.

[0122] 1.4 Account portrait representation:

[0123] The account portrait is formed by the work content portrait, account information portrait and social relationship portrait, realizing the vectorization of the account portrait.

[0124] In summary, the multimodal vectorized representation method of social platform account portraits models the features of the content of works published by platform accounts, account information, and social relationships, and quantizes the features to construct account portraits. By training a large multimodal account portrait representation model, accounts with similar works, information, and social relationships as well as similar portrait features are made to have similar profiles. At the same time, accounts are represented separately in three dimensions by constructing a work content portrait library, an account information portrait library, and a social relationship portrait library. Finally, a unified representation of account portraits is achieved by integrating work content portraits, account information portraits, and social relationship portraits.

[0125] It should be noted that the term "comprises" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article, or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0126] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal vectorized representation method for social platform account portraits, including work content portrait representation, account information portrait representation, social relationship portrait representation and account portrait representation, characterized in that: The specific characterization steps are as follows: 1.1 Characterization of the content of the work: Step 1: Obtaining work information; Step 2: Data preprocessing; Visual text extraction; Speech to text extraction; Video frame extraction; Collection of works texts; Collection of images of works; Step 3: Multimodal feature extraction; Step 4: Vectorize the work content portrait; Step 5: Characterization of the content of the work; 1.2 Account Profile Representation: Step 1: Obtain account information; Step 2: Obtaining work information; Step 3: Data preprocessing; Visual text extraction; Speech to text extraction; Video frame extraction; Collection of works texts; Collection of images of works; Step 4: Historical portrait feature extraction; Step 5: Personal data feature extraction; Step 6: Ethnic feature extraction Semantic features of avatars; Visual semantic features of the work; Semantic features of the work’s text; Step 7: Extract IP location change features; Step 8: Account profile representation; 1.3 Social relationship portrait representation: Step 1: Obtain account relationship; Step 2: Obtain account events; Step 3: Knowledge graph construction; Step 4: Extract social relationship vector; Step 5: Social relationship portrait representation; 1.4 Account portrait representation: The account portrait is formed by the work content portrait, account information portrait and social relationship portrait, realizing the vectorization of the account portrait.

2. According to claim 1, a multi-modal vectorized representation method for social platform account portraits is characterized by: The sampling strategy in the data preprocessing of step 2 of the work content portrait representation is not limited to key frames, frame extraction at fixed time intervals, and extracting content representative frames through a frame extraction algorithm.

3. According to claim 1, a multi-modal vectorized representation method for social platform account portraits is characterized by: The applicable types of cross-modal visual semantic features in the multimodal feature extraction of step three of the work content portrait representation include models with cross-modal visual semantic feature extraction capabilities.

4. According to claim 1, a multi-modal vectorized representation method for social platform account portraits is characterized by: The aggregation methods in the work content portrait vectorization in step 4 of the work content portrait representation include but are not limited to averaging and fusion through a deep learning model.

5. According to claim 1, a multi-modal vectorized representation method for social platform account portraits is characterized by: The sampling strategy in the data preprocessing of step three of the account profile characterization is not limited to key frames, sampling frames at fixed time intervals, and extracting content representative frames through a frame sampling algorithm.

6. A method for multimodal vectorization characterization of social platform account portraits according to claim 1, characterized in that: The applicable types of cross-modal text semantic features in the personal data feature extraction in step five of the account profile representation include models with the ability to extract cross-modal text semantic features.

7. A method for multimodal vectorization characterization of social platform account portraits according to claim 1, characterized in that: The applicable types of cross-modal text semantic features in the ethnic feature extraction of step six of the account profile characterization include models with cross-modal text semantic feature extraction capabilities, and a total of three features, namely, avatar semantic features, visual semantic features, and text semantic features, are used as the ethnic features of the account.

8. The method for multimodal vectorization characterization of social platform account portraits according to claim 1, characterized in that: The graph embedding algorithm in the social relationship vector extraction of step 4 of the social relationship portrait representation refers to an algorithm model with graph embedding capabilities, which is not limited to Node2Vec and DeepWalk.

Images