Method and device for realizing virtual identity association based on graph convolutional neural network of large model and feature enhancement

Abstract

The invention discloses a method and device for realizing virtual identity association based on a large model and a feature enhanced graph convolutional neural network, and the method comprises the following steps: carrying out the fine tuning of a multi-modal large model, and extracting a plurality of attributes with the most discrimination degree of a user, the attributes comprising a user name, posting content, sign-in representation and image information; splicing attributes of the user name, the posting content, the sign-in representation and the image information into a feature matrix, and constructing a neural network structure diagram by using the attributes of the users as nodes, the relationship between the users as edges and the feature matrix as the nodes; the node features are enhanced; identities of different users in the two platforms are embedded into an MLP classifier and output through softmax, and whether the users of the two different platforms belong to the same user or not is judged. According to the method, the virtual identity association model is constructed by using the multi-level attributes of the user, and high-precision user matching is realized under a weak supervision condition through a multi-dimensional feature fusion method.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual identity association, and specifically relates to a method and device for realizing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement. Background Art

[0002] With the rapid development of the Internet and social networks, users' activities on multiple platforms are increasing day by day. This phenomenon has given rise to the need to associate virtual identities across social networks, that is, to identify multiple accounts belonging to the same user on different platforms. This technology is of great significance in personalized recommendations, advertising targeting and network security.

[0003] Traditional virtual identity association methods mainly perform feature extraction and similarity calculation based on three aspects: user attributes, user-generated content, and user social relationships. Specifically, these methods compare the attribute distance between accounts, content statistical features, and the similarity of social graphs, and determine the attribution relationship between accounts by setting thresholds. Although these methods are simple to implement, their accuracy is limited due to the lack of ability to capture complex association patterns. In recent years, with the development of machine learning technology, researchers have begun to use supervised learning methods to associate virtual identities. This type of method can automatically learn classification boundaries from data, but still faces the challenge of feature engineering, which requires manual design of high-quality features.

[0004] Currently, deep learning technology is widely adopted due to its powerful feature representation capabilities. Compared with traditional methods, deep learning can automatically extract multi-level abstract features without human intervention, thereby better capturing the complex correlation characteristics between accounts. However, most existing work only uses information from a single or a few dimensions (such as user attributes or social graphs), which limits the comprehensiveness and accuracy of association analysis. Therefore, how to effectively integrate data from different dimensions and learn consistent representations from them has become a key issue that needs to be solved in this field. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method and device for realizing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement. The present invention utilizes multi-level attributes of users to construct a virtual identity association model, and realizes high-precision user matching under weak supervision conditions through a fusion method of multi-dimensional features.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for associating virtual identities using a graph convolutional neural network based on a large model and feature enhancement, comprising the following steps:

[0008] Fine-tune the multimodal large model and extract multiple attributes of the user that are most discriminative, including username, post content, check-in representation, and image information;

[0009] The attributes of user name, post content, check-in representation and image information are spliced ​​into a feature matrix, and a graph neural network structure diagram is constructed with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes;

[0010] After the neural network structure diagram is obtained, the neighbor nodes are screened, and the nodes with small out-degree and in-degree are blocked to retain those nodes with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplements for the target nodes, and these supplementary information are weighted and integrated through the multi-head attention mechanism to further enhance the feature representation and information richness of the target nodes;

[0011] The identities of different users in the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users of two different platforms belong to the same user.

[0012] As a preferred technical solution, the method for extracting the user name is as follows:

[0013] Divide the user name into a unique character sequence according to the characters, wherein the character sequence includes characters and numerical values, and the numerical values ​​represent the character count of the corresponding characters in the user name;

[0014] Get a count weighted vector of the user name based on the character sequence;

[0015] The feature vector representation of the final user name is obtained based on the preset single-layer autoencoder.

[0016] As a preferred technical solution, the method for extracting the post content is as follows:

[0017] Embed the user post text Corr_Text and the corresponding image Corr_image of the post into the preset prompt template;

[0018] The above prompt template is used as the input of the pre-trained large language model, so that the large language model can accurately extract the text features and image features Z of the post;

[0019] Z=LLM C (Prompt,corr_Text,corr_image);

[0020] Among them, Z is composed of the large language model LLM CThe output combines the multimodal information of the post text and the corresponding image, representing the overall characteristics of the post; Z contains the semantic information extracted from the text description and the visual features extracted from the image. These features are processed and fused by the large language model, and the generated Z is a comprehensive description of the post;

[0021] Z is further processed using the pre-trained BERT model to extract deep text features.

[0022] As a preferred technical solution, the check-in representation extraction is specifically as follows:

[0023] Select to construct a spatiotemporal feature matrix to realize the modeling of user check-in information based on time periods. The dimension of the spatiotemporal feature matrix is ​​D L ×D T , at the (m,n) position of the spatiotemporal feature matrix, if the value is 1, it means that the user has arrived at location n within the time period m; accordingly, according to the check-in location and check-in time of the check-in post, the corresponding position in the spatiotemporal feature matrix is ​​set to 1, indicating the user's check-in;

[0024] For users who have not posted any sign-in posts, the IP address is used to fill in the missing information; the corresponding position of the spatiotemporal feature matrix is ​​set to 1 for sign-in for the collection time period and its address;

[0025] The spatiotemporal feature matrix is ​​processed by ST-GCN to obtain the final sign-in representation.

[0026] As a preferred technical solution, the image information extraction is specifically as follows:

[0027] Extract image features based on residual network;

[0028] Perform object detection on user-uploaded images based on the YOLOv9 algorithm and match the detected objects with images in the local database. The matching process is defined as finding the database item that is most similar to the detected object in the geo-tag and label space.

[0029] Extract the geotags of all users, encode them using the pre-trained BERT model, and convert them into embedding vectors.

[0030] As a preferred technical solution, the feature representation of the enhanced target node is specifically as follows:

[0031] User a's social connections are n means that user a has n social connections in total. Since not every social connection has the same importance, the topological importance score s of the neighbor nodes is defined as i , where i represents the node index, and degree centrality or other graph centrality indicators are used to calculate s i,Right now:

[0032]

[0033] Where ∈ is a small constant used to prevent division by zero errors, and degree(i) represents the degree of node i, that is, the number of nodes directly connected to node i;

[0034] Design a multi-head attention layer, where each head calculates the influence of neighbor nodes on the target node. The influence of node j on node i is calculated using the following formula:

[0035]

[0036] Where W (k) is the weight matrix of the head, a (k) is the attention vector, ∥ indicates vector connection;

[0037] According to s i The value of Adjustments are made to suppress nodes with low topological importance, which can be expressed as follows:

[0038]

[0039] Updating node features using inhibitory multi-head attention.

[0040] As a preferred technical solution, the identities of different users on the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users on the two different platforms belong to the same user, specifically:

[0041] For any two user nodes i and j, their feature vectors h i and h j Pair them to form a joint feature vector h ij ;

[0042] Use a multi-layer perceptron MLP to process the joint feature vector h ij ; MLP contains multiple hidden layers and nonlinear activation functions to enhance the model's ability to handle nonlinear data, and finally performs classification through an output layer:

[0043] z ij =σ(W 2 ReLU(W 1 h ij +b 1 )+b 2 )

[0044] Where W 1 ,W 2 and b 1 ,b2 is the weight matrix and bias vector of the MLP, σ is the sigmoid activation function of the output layer, which is used to normalize the output to between 0 and 1, that is, to output the probability that two nodes belong to the same natural person;

[0045] The probability value z based on the MLP output ij The value of z determines whether two users belong to the same natural person; by setting a threshold θ, if z ij ≥θ, then the two users are judged to be the same person;

[0046]

[0047] When z ij When ≥θ, the function outputs 1, indicating that according to the model, users i and j are the same natural person;

[0048] When z ij When <θ, the function outputs 0, indicating that users i and j are not the same natural person.

[0049] In the second aspect, the present invention proposes a virtual identity association system based on a large model and feature enhanced graph convolutional neural network, which is applied to the virtual identity association method based on a large model and feature enhanced graph convolutional neural network, including a feature extraction module, a feature fusion module, a node feature supplement module, and an identity association module;

[0050] The feature extraction module is used to extract multiple attributes of the user that are most distinguishable, including user name, post content, check-in representation, and image information;

[0051] The feature fusion module is used to stitch the attributes of the user name, post content, check-in representation and image information into a feature matrix, and construct a neural network structure diagram with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes;

[0052] The node feature supplementation module is used to screen neighbor nodes after the graph neural network structure diagram, screen out nodes with small out-degree and in-degree, and retain those nodes with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplements for the target node, and these supplementary information are weighted and integrated through a multi-head attention mechanism to further enhance the feature representation and information richness of the target node;

[0053] The identity association module is used to embed the identities of different users on the two platforms into the MLP classifier and output through softmax to determine whether the users on the two different platforms belong to the same user.

[0054] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0055] at least one processor; and,

[0056] a memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method for realizing virtual identity association based on the graph convolutional neural network with large model and feature enhancement.

[0058] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the method for associating virtual identities using a graph convolutional neural network based on a large model and feature enhancement.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] The present invention uses the multi-level attributes of users to construct a virtual identity association model, including user feature extraction, feature enhancement, and network structure diagram, large model fine-tuning and other processes. The fusion method of multi-dimensional features of the present invention can achieve high-precision user matching under weak supervision conditions, which is often not done in the prior art. And the present invention uses a graph convolutional network (GCN) to process complex social network data, and optimizes the extraction and processing of information through a multi-head attention mechanism. This method can effectively process heterogeneous and multimodal data and enhance the learning and representation capabilities of the model. In the network structure diagram, using an inhibitory multi-head attention mechanism, giving priority to nodes with high topological importance, this design can improve the accuracy and efficiency of association analysis, and can greatly alleviate the imbalance of user information resources on different platforms and cause low association success rates. And the use of fine-tuning large models can better achieve information association complementarity of different modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 A flowchart of a method for associating virtual identities based on a large model and a feature-enhanced graph convolutional neural network according to an embodiment of the present invention;

[0063] Figure 2 This is a block diagram of a virtual identity association system implemented based on a large model and a feature-enhanced graph convolutional neural network according to an embodiment of the present invention.

[0064] Figure 3 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0066] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0067] like Figure 1 As shown, the present invention provides a method for realizing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement, comprising the following steps:

[0068] S1. Extract multiple attributes of the user that are most distinctive, including user name, post content, check-in representation, and image information.

[0069] S1.1, Username extraction;

[0070] Usernames are the data that best represents user characteristics. Sufficient surveys have proven that users are accustomed to using the same or similar usernames on different platforms. Since usernames can contain a variety of characters such as letters, numbers, and special symbols, in order to fully extract username features, this embodiment embeds character-level attributes for usernames to ensure text similarity between similar users.

[0071] Step S1.1 is specifically as follows:

[0072] S1.1.1. In this embodiment, the user name can be divided into unique character sequences according to the characters. For example, "abandon_" can be divided into w = 'a': 2, 'b': 1, 'c': 0, 'd': 1, ..., 'n': 2, ..., 'o': 1, ..., others: 1. The numerical value represents the character count of the corresponding character in the text, and others represents all punctuation marks except numbers and English characters. Therefore, the count weighted vector of the user name can be expressed as c i =[2,1,0,1…,2…,1,…1]; In order to reduce the impact of excessively large dimensions in deep learning, this embodiment uses a single-layer autoencoder to embed the count weighted vector into the distributed representation.

[0073] S1.1.2. This embodiment defines an autoencoder that maps an input vector x to a hidden representation z through a deterministic function z=f(x)=Wx+b. Through a decoding function y=g(z)=W * z+b * Restore the hidden feature representation z; the formula is as follows:

[0074] z=f(x)=Wx+b,

[0075] y=g(z)=W * z+b * ,

[0076] in, and are the weight matrix and bias vector respectively. is the weight matrix of the decoding function g(·).

[0077] The loss function of the autoencoder is defined as:

[0078] L=∑∥y i -c i ∥ 2

[0079] The encoder parameters W are trained by minimizing the loss function e ,b e , thus obtaining the reduced-dimensional user name feature representation X c :

[0080] X ↓ =W e c+b e .

[0081] S1.2, post content extraction;

[0082] In terms of user post feature extraction, this embodiment innovatively adopts the method of large language model (LLM) fine-tuning and customized prompt design to perform multimodal comprehensive analysis of post features. Through the customized prompt template, this embodiment can efficiently generate text descriptions and image features of posts. The specific design is as follows:

[0083] S1.2.1. Use multimodal image-text datasets to fine-tune large models to improve the comprehensive representation capabilities of the two modalities.

[0084] S1.2.2 Prepare a dataset of text, images and their comprehensive representations, and construct the following data format:

[0085]

[0086] S1.2.3. Use instruction fine-tuning to improve the ability of large models to fine-tune professional fields, thereby improving their multi-modal comprehensive representation capabilities.

[0087] S1.2.4. Set the Prompt template to instruct the model to generate a text description of the post; the Prompt template is as follows: Prompt: {

[0088] ' <instuction>Generate a text description within 100characters of thetext and image of the social media post,describing it from a first-personperspective<\instruction>\n′

[0089] ′ <text>{{Text}}<\Text>\n′

[0090] ' {{Corr_image}}\n′}

[0091] S1.2.5, LLM C Indicates the output of the large model; embed the user post text Corr_Text and the corresponding image Corr_image of the post into prompt. Through this structured instruction, it can ensure that the large language model accurately extracts the text and image features of the post, as follows:

[0092] Z=LLM C (Prompt,corr_Text,corr_image)

[0093] Among them, Z is generated by the Large Language Model (LLM), which combines the multimodal information of the post text and image and represents the overall characteristics of the post. Specifically, Z contains the semantic information extracted from the text description and the visual features extracted from the image. These features are processed and fused by the large language model, and the generated Z is a comprehensive description of the post.

[0094] S1.2.6. Use the BERT model to further process Z and extract its deep text features. BERT (Bidirectional Encoder Representation Transformation) is a pre-trained language model that can capture rich semantic and contextual information in the text and generate high-quality feature representations. The extraction process is as follows:

[0095] X w =BERT(Z)

[0096] S1.3, Check-in means extraction:

[0097] Check-in posts can reflect the user's life journey, which is crucial to user identity association. In order to represent the user's check-in attributes, the common method is one-hot encoding, which checks in for each user's check-in location. However, this method ignores the importance of the time dimension. Even if two users check in at the same place, they may be misclassified as the same user if their check-in times are different.

[0098] Therefore, this embodiment chooses to construct a spatiotemporal feature matrix to realize the modeling of user check-in information based on time periods. Specifically, this embodiment divides time by month and sets the dimension D T The locations are processed by country. Since the virtual identity is associated with a domestic account, the check-in points in China are divided by city, and the check-in points outside China are divided by country. For example, if user B has checked in in Beijing and Bangkok, Thailand, he will only be marked as having checked in in Beijing, China and Thailand. The final dimension is D L The sequence listing of .

[0099] Because the dimension of the spatiotemporal feature matrix E is D L ×D T , at the (m,n) position of the feature matrix, if the value is 1, it means that the user has been to location n within the time period m. Accordingly, according to the check-in location and check-in time of the check-in post, the corresponding position in the feature matrix E is set to 1, indicating the user's check-in.

[0100] For users who have not signed in to post, the present invention collects IP addresses to fill in missing information. Some social platforms require IP address updates, so IP address collection can be performed for these social platforms. Set 1 to sign in at the corresponding position of the space-time matrix E for the collection time period and its address.

[0101] Since the address and time period sequences are large, the feature dimension of the sign-in matrix is ​​too large, so ST-GCN is used for dimensionality reduction. A large number of experiments and papers have proved that the spatiotemporal convolutional network (ST-GCN) is a neural network model that can efficiently process data with spatiotemporal structures. Therefore, the feature matrix E is processed by ST-GCN. The final sign-in representation can be obtained by the following formula:

[0102]

[0103] E=(T∪L ip )

[0104] S1.4, Image information extraction: For images posted by users, two schemes are adopted for parallel processing:

[0105] S1.4.1. RESNET performs image feature extraction; the details are as follows:

[0106] Users are accustomed to posting their favorite images on different social media, which helps to achieve virtual identity association across different platforms. For users' visual posts, residual networks (ResNet) are used for encoding. Through this architecture, users' visual posts can be converted into a set of complex feature vectors, which not only contain the original information of the image, but also contain deep semantic and style features.

[0107] In order to further refine and optimize the user's visual representation, this embodiment designs a unique mathematical model that combines pooling technology, nonlinear activation functions, and fully connected network layers. This model can not only process high-dimensional data, but also achieve dimensionality reduction and abstraction of features while maintaining information integrity.

[0108] This embodiment defines the following mathematical formula to formalize this process:

[0109] V=f avg (R(V 1 ),R(V 2 ),...,R(V n ))

[0110] V=g(W·f avg (V)+b)

[0111] Among them, V represents the optimized user visual feature vector, V i represents the feature encoding of the i-th visual post, R represents the encoding function of the ResNeT network, and f avg represents an advanced average pooling function used to integrate the features of multiple visual posts, g is a nonlinear activation function used to introduce nonlinear characteristics and enhance the expressiveness of the model, and W and b represent the weight matrix and bias vector of the fully connected layer, respectively.

[0112] S1.4.2, check-in attractions database identification;

[0113] Check-in attraction database identification: users are accustomed to checking in local attractions and uploading them to social networks. The present invention uses the most advanced and fastest YOLOV9 algorithm to detect objects in images and matches them with the local database "OpenImages Dataset", which contains a large dataset of millions of images, many of which are geotagged. This dataset also contains rich label information, which is suitable for this visual task.

[0114] Use the YOLOv9 algorithm to detect objects on images uploaded by users; let the input image be I, the object detection algorithm outputs a set of bounding boxes and category labels Among them B i represents the bounding box of the i-th target, C i Represents its category label.

[0115]

[0116] Among them, Θ YOLOv9 are the parameters of the YOLOv9 model.

[0117] Match the detected targets with the images in the local database Open Images Dataset; let the images in the database and their geo-tags and label information be Among them G j Represents image I j Geographical marking, C j Indicates its label information.

[0118] The matching process can be defined as finding the database item that is most similar to the detected object in geotag and label space:

[0119]

[0120] Among them, Sim(·,·) is a similarity measurement function used to measure the similarity between the detected target and the database item. argmax() is used to extract the matching item with the highest similarity.

[0121] Finally, extract the geographic tags G of all users j , if the user image contains Beijing, China, Shenzhen, China, Bangkok, Thailand, and Seoul, South Korea, then a user image list U is constructed i = [Beijing, China, Shenzhen, China, Bangkok, Thailand, Seoul, South Korea]. Next, the pre-trained BERT model is used to encode the geographic tags and convert them into embedded vectors. The specific process is as follows:

[0122] The pre-trained BERT model is used to encode the text sequence of each geo-tag and generate the corresponding embedding vector. Let the geo-tag list be U i =[u 1 ,u 2 ,…,u n ], through the encoding process of the BERT model, the embedding vector E of each geographic tag can be obtained i =[e 1 ,e 2 ,…,e n ], where each e k Indicates geographic tag u k high-dimensional embedding vector of .

[0123] For each geographic tag u k , whose embedding vector e k Obtained through the encoding process of the BERT model

[0124] e k =BERT(u k |Θ BERT )

[0125] Among them, Θ BERT Represents the pre-trained parameters of the BERT model.

[0126] S2. The attributes of the user name, post content, check-in representation, and image information are spliced ​​into a feature matrix, and a graph neural network structure diagram is constructed with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes.

[0127] S3, after the graph neural network structure diagram, the neighbor nodes will be screened and the nodes with small out-degree and in-degree will be blocked. In this way, only those nodes with significant influence in the network are retained, thereby enhancing the model's ability to capture important node relationships. At the same time, the features of the retained neighbor nodes are used as information supplements for the target node, and these supplementary information are weighted and integrated through the multi-head attention mechanism to further enhance the feature representation and information richness of the target node.

[0128] After obtaining the heterogeneous multimodal representation and social connection representation of each user, we should continue to study how to seamlessly integrate these representations to achieve accurate user representation learning. Since the user's social topology structure has been obtained, it is beneficial for the present invention to use neighbor node information to supplement user information.

[0129] The present invention adopts graph convolutional network (GCN) to solve the problem of cross-social network virtual identity association. GCN has significant advantages in processing complex network data, and can effectively utilize users' connection patterns and interaction information on different social platforms, thereby improving the accuracy and robustness of identity association. GCN encodes and splices the features of each user node (such as user name, IP address, published text, etc.), and converts these features into vector representations in high-dimensional space. This process uses the multiplication of the adjacency matrix and the feature matrix to achieve deep feature extraction through a nonlinear transformation layer.

[0130] The user name, post, check-in and image attributes are concatenated into a feature matrix E, with users as nodes and relationships between users as edges. The social connections of user a are n represents that user a has n social connections in total. Since not every social connection has the same importance, the present invention defines the topological importance score s of neighbor nodes: i , where i represents the node index. Degree Centrality or other graph centrality metrics are used to calculate s i ,Right now:

[0131]

[0132] Where ∈ is a small constant used to prevent division by zero errors. degree(i) represents the degree of node i, that is, the number of nodes directly connected to node i.

[0133] At the same time, a multi-head attention layer is designed, and each head calculates the influence of neighbor nodes on the target node. The influence of node j to node i is calculated by the following formula:

[0134]

[0135] Where W (k) is the weight matrix of the head, a (k) is the attention vector, and ∥ indicates vector connection.

[0136] According to s i The value of Adjustments are made to suppress nodes with low topological importance, which can be expressed as

[0137]

[0138] This step ensures that the model prioritizes nodes with high topological importance when aggregating neighbor node features.

[0139] Finally, the node features are updated using inhibitory multi-head attention:

[0140]

[0141] Where σ is the nonlinear activation function and K is the number of attention heads.

[0142] S4. The identities of different users on the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users on the two different platforms belong to the same user.

[0143] The previous module uses a graph neural network, and each user account corresponds to a node in the social network graph. Through the above method, each node i obtains a comprehensive feature representation h i For any two user nodes i and j, their feature vectors h i and h j Pairing is performed to form a joint feature vector. Common pairing methods include vector connection, vector difference or vector dot product, etc. The present invention adopts vector connection, that is:

[0144]

[0145] in Representation vector concatenation operation. This operation aims to combine the information in two feature vectors to build a comprehensive joint feature representation.

[0146] Use a multilayer perceptron to process the joint feature vector h ij MLP contains multiple hidden layers and non-linear activation functions to enhance the model's ability to handle non-linear data, such as ReLU, and finally an output layer for classification:

[0147] z ij =σ(W 2 ReLU(W 1 h ij +b 1 )+b 2 )

[0148] Where W 1 ,W 2 and b 1 ,b 2 is the weight matrix and bias vector of the MLP, and σ is the sigmoid activation function of the output layer, which is used to normalize the output to between 0 and 1, that is, to output the probability that two nodes belong to the same natural person.

[0149] The probability value z based on the MLP output ij The value of z determines whether two users belong to the same natural person. Usually, by setting a threshold θ, if z ij ≥θ, then the two users are judged to be the same person.

[0150]

[0151] ·When ij When ≥θ, the function outputs 1, indicating that according to the model, users i and j are the same natural person.

[0152] ·When ij When <θ, the function outputs 0, indicating that users i and j are not the same natural person.

[0153] The present invention proposes a method for constructing a virtual identity association model using multi-level attributes of users, which can achieve highly accurate user matching in a weakly supervised environment. By integrating features of multiple dimensions, the present invention enhances the accuracy of user identity recognition.

[0154] Specifically, this technology uses a graph convolutional network (GCN) to process complex social network data. Graph convolutional networks can effectively process heterogeneous and multimodal data, thereby enhancing the learning and representation capabilities of the model. In addition, by introducing a multi-head attention mechanism, the present invention further optimizes the information extraction and processing process.

[0155] At the network structure level, the present invention utilizes a multi-head attention mechanism to identify and prioritize nodes with higher topological importance in social networks, which helps to improve the accuracy and processing efficiency of association analysis.

[0156] In short, the present invention provides an efficient and accurate method to process complex data sets and achieve effective matching of user identities by combining a graph convolutional network with a multi-head attention mechanism.

[0157] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0158] Based on the same idea as the method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement in the above-mentioned embodiment, the present invention also provides a system for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement, which can be used to execute the above-mentioned method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement. For ease of explanation, in the structural schematic diagram of the embodiment of the system for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement, only the parts related to the embodiment of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or a combination of certain components, or different component arrangements.

[0159] See also Figure 2 In another embodiment of the present application, a virtual identity association system 100 is provided based on a large model and feature enhanced graph convolutional neural network, the system comprising a feature extraction module 101, a feature fusion module 102, a node feature supplementation module 103, and an identity association module 104;

[0160] The feature extraction module 101 is used to extract multiple attributes of the user that are most distinguishable, including user name, post content, check-in representation, and image information;

[0161] The feature fusion module 102 is used to stitch the attributes of user name, post content, check-in representation and image information into a feature matrix, and construct a graph neural network structure diagram with users as nodes, relationships between users as edges, and feature matrices as node attributes; the node feature supplement module 103 is used to filter neighbor nodes after the graph neural network structure diagram, screen out nodes with small out-degree and in-degree, and retain those nodes with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplements for the target node, and these supplementary information are weighted and integrated through a multi-head attention mechanism to further enhance the feature representation and information richness of the target node.

[0162] The identity association module 104 is used to embed the identities of different users on the two platforms into the MLP classifier and output through softmax to determine whether the users on the two different platforms belong to the same user.

[0163] It should be noted that the virtual identity association system based on a large model and feature enhanced graph convolutional neural network of the present invention corresponds one to one with the virtual identity association method based on a large model and feature enhanced graph convolutional neural network of the present invention. The technical features and beneficial effects described in the embodiment of the virtual identity association method based on a large model and feature enhanced graph convolutional neural network are applicable to the embodiment of the virtual identity association based on a large model and feature enhanced graph convolutional neural network. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here.

[0164] In addition, in the implementation of the virtual identity association system based on the large model and feature enhanced graph convolutional neural network in the above embodiment, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the convenience of software implementation due to the configuration requirements of the corresponding hardware. That is, the internal structure of the virtual identity association system based on the large model and feature enhanced graph convolutional neural network is divided into different program modules to complete all or part of the functions described above.

[0165] See also Figure 3 In one embodiment, an electronic device for implementing a method for associating a virtual identity based on a graph convolutional neural network with a large model and feature enhancement is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a program 203 for associating a virtual identity based on a graph convolutional neural network with a large model and feature enhancement.

[0166] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the virtual identity association program 203 based on the large model and feature enhancement graph convolutional neural network, but also can be used to temporarily store data that has been output or is to be output.

[0167] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202.

[0168] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0169] The virtual identity association program 203 based on a large model and feature-enhanced graph convolutional neural network stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0170] Extracting multiple attributes of the user that are most distinctive, including user name, post content, check-in representation, and image information;

[0171] The attributes of user name, post content, check-in representation, and image information are spliced ​​into a feature matrix, and the graph neural network structure diagram is constructed with users as nodes, relationships between users as edges, and feature matrices as node attributes; after the graph neural network structure diagram, we will filter the neighbor nodes and block out nodes with small out-degree and in-degree. In this way, only those nodes with significant influence in the network are retained, thereby enhancing the model's ability to capture important node relationships. At the same time, the features of the retained neighbor nodes are used as information supplements for the target node, and these supplementary information are weighted and integrated through a multi-head attention mechanism to further enhance the feature representation and information richness of the target node.

[0172] The identities of different users in the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users of two different platforms belong to the same user.

[0173] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0175] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.< / text> < / instuction>

Claims

1. A virtual identity association method based on a large model and feature-enhanced graph convolutional neural network, characterized in that: The steps include: Fine-tune the multimodal large model and extract multiple attributes of the user that are most discriminative, including username, post content, check-in representation, and image information; The attributes of user name, post content, check-in representation and image information are spliced ​​into a feature matrix, and a graph neural network structure diagram is constructed with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes; After the neural network structure diagram is obtained, the neighbor nodes are screened, and the nodes with small out-degree and in-degree are blocked to retain those nodes with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplements for the target nodes, and these supplementary information are weighted and integrated through the multi-head attention mechanism to further enhance the feature representation and information richness of the target nodes; The identities of different users in the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users of two different platforms belong to the same user.

2. According to claim 1, the method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement is characterized in that: The method for extracting the user name is as follows: Divide the user name into a unique character sequence according to the characters, wherein the character sequence includes characters and numerical values, and the numerical values ​​represent the character count of the corresponding characters in the user name; Get a count weighted vector of the user name based on the character sequence; The feature vector representation of the final user name is obtained based on the preset single-layer autoencoder.

3. The method for associating virtual identities based on a graph convolutional neural network with a large model and feature enhancement according to claim 1 is characterized in that: The method for extracting the post content is as follows: Embed the user post text Corr_Text and the corresponding image Corr_image of the post into the preset prompt template; The above prompt template is used as the input of the pre-trained large language model, so that the large language model can accurately extract the text features and image features Z of the post; Z=LLM C (Prompt,corr_Text,corr_image); Among them, Z is composed of the large language model LLM C The output combines the multimodal information of the post text and the corresponding image, representing the overall characteristics of the post; Z contains the semantic information extracted from the text description and the visual features extracted from the image. These features are processed and fused by the large language model, and the generated Z is a comprehensive description of the post; Z is further processed using the pre-trained BERT model to extract deep text features.

4. According to claim 1, the method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement is characterized in that: The check-in representation extraction is specifically as follows: Select to construct a spatiotemporal feature matrix to realize the modeling of user check-in information based on time periods. The dimension of the spatiotemporal feature matrix is ​​D L ×D T , at the (m,n) position of the spatiotemporal feature matrix, if the value is 1, it means that the user has arrived at location n within the time period m; accordingly, according to the check-in location and check-in time of the check-in post, the corresponding position in the spatiotemporal feature matrix is ​​set to 1, indicating the user's check-in; For users who have not posted any sign-in posts, the IP address is used to fill in the missing information; the corresponding position of the spatiotemporal feature matrix is ​​set to 1 for sign-in for the collection time period and its address; The spatiotemporal feature matrix is ​​processed by ST-GCN to obtain the final sign-in representation.

5. According to claim 1, the method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement is characterized in that: The image information extraction is specifically as follows: Extract image features based on residual network; Perform object detection on user-uploaded images based on the YOLOv9 algorithm and match the detected objects with images in the local database. The matching process is defined as finding the database item that is most similar to the detected object in the geo-tag and label space. Extract the geotags of all users, encode them using the pre-trained BERT model, and convert them into embedding vectors.

6. The method for associating virtual identities using a graph convolutional neural network based on a large model and feature enhancement according to claim 1, characterized in that: The feature representation of the enhanced target node is specifically: User a's social connections are n means that user a has n social connections in total. Since not every social connection has the same importance, the topological importance score s of the neighbor nodes is defined as i , where i represents the node index, and degree centrality or other graph centrality indicators are used to calculate s i ,Right now: Where ∈ is a small constant used to prevent division by zero errors, and degree(i) represents the degree of node i, that is, the number of nodes directly connected to node i; Design a multi-head attention layer, where each head calculates the influence of neighbor nodes on the target node. The influence of node j on node i is calculated using the following formula: Where W (k) is the weight matrix of the head, a (k) is the attention vector, ∥ indicates vector connection; According to s i The value of Adjustments are made to suppress nodes with low topological importance, which can be expressed as follows: Updating node features using inhibitory multi-head attention.

7. The method for associating virtual identities based on a graph convolutional neural network with a large model and feature enhancement according to claim 1, characterized in that: The identities of different users on the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users on the two different platforms belong to the same user, specifically: For any two user nodes i and j, their feature vectors h i and h j Pair them to form a joint feature vector h ij ; Use a multi-layer perceptron MLP to process the joint feature vector h ij ; MLP contains multiple hidden layers and nonlinear activation functions to enhance the model's ability to handle nonlinear data, and finally performs classification through an output layer: With ij =σ(W2 ReLU(W1h ij +b1)+b2) Where W1, W2 and b1, b2 are the weight matrix and bias vector of MLP, σ is the sigmoid activation function of the output layer, which is used to normalize the output to between 0 and 1, that is, to output the probability that two nodes belong to the same natural person; The probability value z based on the MLP output ij The value of z determines whether two users belong to the same natural person; by setting a threshold θ, if z ij ≥θ, then the two users are judged to be the same person; When z ij When ≥θ, the function outputs 1, indicating that according to the model, users i and j are the same natural person; When z ij When <θ, the function outputs 0, indicating that users i and j are not the same natural person.

8. A virtual identity association system is implemented based on a large model and feature-enhanced graph convolutional neural network, which is characterized by: A method for realizing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement as described in any one of claims 1 to 7, comprising a feature extraction module, a feature fusion module, a node feature supplementation module, and an identity association module; The feature extraction module is used to extract multiple attributes of the user that are most distinguishable, including user name, post content, check-in representation, and image information; The feature fusion module is used to stitch the attributes of the user name, post content, check-in representation and image information into a feature matrix, and construct a neural network structure diagram with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes; The node feature supplement module is used to screen neighbor nodes after the graph neural network structure diagram, screen out nodes with small out-degree and in-degree, and retain those nodes with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplements for the target nodes, and these supplementary information are weighted and integrated through a multi-head attention mechanism to further enhance the feature representation and information richness of the target nodes; the identity association module is used to embed the identities of different users on the two platforms into the MLP classifier and output through softmax to determine whether users on two different platforms belong to the same user.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method for realizing virtual identity association based on a large model and feature enhanced graph convolutional neural network as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the method for associating virtual identities by using a graph convolutional neural network based on a large model and feature enhancement as described in any one of claims 1 to 7 is implemented.

Images