Method and device for virtual identity association based on graph convolutional neural network with large model and feature enhancement

Abstract

The present invention discloses a method and device for implementing virtual identity association using a graph convolutional neural network based on a large model and feature enhancement. The method includes the following steps: fine-tuning a large multimodal model and extracting multiple user attributes with the highest degree of discrimination, including username, post content, check-in representation, and image information; concatenating the attributes of username, post content, check-in representation, and image information into a feature matrix, constructing a graph neural network structure diagram with users as nodes, relationships between users as edges, and feature matrices as node attributes; enhancing node features; and embedding the identities of different users on two platforms into an MLP classifier and outputting them through softmax to determine whether users on two different platforms belong to the same user. The present invention utilizes multi-level user attributes to construct a virtual identity association model. By fusing multi-dimensional features, high-precision user matching is achieved under weak supervision.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual identity association, and specifically relates to a method and device for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement. Background Art

[0002] With the rapid development of the internet and social networks, users are increasingly active across multiple platforms. This phenomenon has given rise to the need to link virtual identities across social networks—that is, to identify multiple accounts belonging to the same user across different platforms. This technology is crucial for personalized recommendations, ad targeting, and network security.

[0003] Traditional virtual identity association methods primarily extract features and calculate similarities based on three aspects: user attributes, user-generated content, and user social relationships. Specifically, these methods compare attribute distances, content statistical features, and social graph similarities between accounts, and determine the attribution between accounts by setting thresholds. While simple to implement, these methods are limited in accuracy due to their lack of ability to capture complex association patterns. In recent years, with the advancement of machine learning technology, researchers have begun to employ supervised learning methods for virtual identity association. These methods can automatically learn classification boundaries from data, but they still face the challenge of feature engineering, which requires the manual design of high-quality features.

[0004] Deep learning technology is currently being widely adopted due to its powerful feature representation capabilities. Compared to traditional methods, deep learning can automatically extract multi-level abstract features without human intervention, thereby better capturing the complex associations between accounts. However, most existing work only utilizes information from a single or a few dimensions (such as user attributes or social graphs), which limits the comprehensiveness and accuracy of association analysis. Therefore, how to effectively integrate data from different dimensions and learn consistent representations from them has become a key issue that needs to be addressed in this field. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method and device for realizing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement. The present invention utilizes the multi-level attributes of users to construct a virtual identity association model, and realizes high-precision user matching under weak supervision conditions through the fusion method of multi-dimensional features.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for associating virtual identities using a graph convolutional neural network based on a large model and feature enhancement, comprising the following steps:

[0008] Fine-tune the multimodal large model and extract the most discriminative attributes of users, including username, post content, check-in information, and image information;

[0009] The attributes of user name, post content, check-in representation, and image information are spliced ​​into a feature matrix, and a neural network structure diagram is constructed with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes;

[0010] After the neural network structure diagram is drawn, the neighbor nodes are screened, and the nodes with small out-degree and in-degree are blocked to retain those with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplement for the target node, and these supplementary information are weighted and integrated through the multi-head attention mechanism to further enhance the feature representation and information richness of the target node;

[0011] The identities of different users on the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users on two different platforms belong to the same user.

[0012] As a preferred technical solution, the method for extracting the username is as follows:

[0013] Divide the user name into a unique character sequence based on the characters, wherein the character sequence includes characters and numerical values, and the numerical values ​​represent the character count of the corresponding characters in the user name;

[0014] Get the count weighted vector of the username based on the character sequence;

[0015] The feature vector representation of the final user name is obtained based on the preset single-layer autoencoder.

[0016] As a preferred technical solution, the method for extracting the post content is as follows:

[0017] Embed the user post text Corr_Text and the corresponding image Corr_image of the post into the preset prompt template;

[0018] The above prompt template is used as the input of the pre-trained large language model, so that the large language model can accurately extract the text features and image features Z of the post;

[0019] Z=LLM C (Prompt,corr_Text,corr_image);

[0020] Among them, Z is composed of the large language model LLM CThe output combines multimodal information from the post text and the corresponding image, representing the overall characteristics of the post. Z contains semantic information extracted from the text description and visual features extracted from the image. These features are processed and integrated by a large language model to generate Z, which is a comprehensive description of the post.

[0021] Z is further processed using the pre-trained BERT model to extract deep text features.

[0022] As a preferred technical solution, the check-in representation extraction is specifically as follows:

[0023] Select and construct a spatiotemporal feature matrix to realize the modeling of user check-in information based on time period. The spatiotemporal feature matrix dimension is D L ×D T , at the (m,n) position of the spatiotemporal feature matrix, if the value is 1, it means that the user has arrived at location n within the time period m; accordingly, according to the check-in location and check-in time of the check-in post, the corresponding position in the spatiotemporal feature matrix is ​​set to 1, indicating the user's check-in;

[0024] For users who have not posted a sign-in post, the IP address is used to fill in the missing information; the corresponding position of the spatiotemporal feature matrix of the collection time period and its address is set to 1 for sign-in;

[0025] The spatiotemporal feature matrix is ​​processed by ST-GCN to obtain the final sign-in representation.

[0026] As a preferred technical solution, the image information extraction is specifically as follows:

[0027] Extract image features based on residual network;

[0028] Perform object detection on user-uploaded images based on the YOLOv9 algorithm and match the detected objects with images in the local database. The matching process is defined as finding the database item that is most similar to the detected object in the geotag and tag space.

[0029] Extract the geotags of all users, encode them using the pre-trained BERT model, and convert them into embedding vectors.

[0030] As a preferred technical solution, the feature representation of the enhanced target node is specifically as follows:

[0031] User a's social connections are n represents that user a has n social connections. Since not every social connection has the same importance, the topological importance score s of the neighbor nodes is defined as i , where i represents the node index, and degree centrality or other graph centrality indicators are used to calculate s i,Right now:

[0032]

[0033] Where ∈ is a small constant used to prevent division by zero errors, and degree(i) represents the degree of node i, that is, the number of nodes directly connected to node i;

[0034] Design a multi-head attention layer, where each head calculates the influence of neighboring nodes on the target node. The influence of node j on node i is calculated using the following formula:

[0035]

[0036] Where W (k) is the weight matrix of the head, a (k) is the attention vector, ∥ represents vector connection;

[0037] According to s i The value of Adjustments are made to suppress nodes with low topological importance, specifically expressed as follows:

[0038]

[0039] Updating node features using inhibitory multi-head attention.

[0040] As a preferred technical solution, the identities of different users on the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users on the two different platforms belong to the same user. Specifically:

[0041] For any two user nodes i and j, their feature vectors h i and h j Pair them to form a joint feature h ij ;

[0042] Use a multi-layer perceptron (MLP) to process the joint feature vector h ij ; MLP contains multiple hidden layers and nonlinear activation functions to enhance the model's ability to handle nonlinear data, and finally performs classification through an output layer:

[0043] z ij =σ(W2·ReLU(W1h ij +b1)+b2)

[0044] Where W1, W2 and b1, b2 are the weight matrix and bias vector of the MLP, and σ is the sigmoid activation function of the output layer, which is used to normalize the output to between 0 and 1, that is, to output the probability that two nodes belong to the same natural person;

[0045] The probability value z based on the MLP output ij The value of z determines whether two users belong to the same natural person; by setting a threshold θ, if z ij ≥θ, then the two users are judged to be the same person;

[0046]

[0047] When z ij When ≥θ, the function outputs 1, indicating that according to the model, users i and j are the same natural person;

[0048] When z ij When <θ, the function outputs 0, indicating that users i and j are not the same natural person.

[0049] In a second aspect, the present invention proposes a virtual identity association system based on a large model and feature enhancement graph convolutional neural network, which is applied to the virtual identity association method based on a large model and feature enhancement graph convolutional neural network, including a feature extraction module, a feature fusion module, a node feature supplementation module, and an identity association module;

[0050] The feature extraction module is used to extract multiple attributes of the user that are most distinctive, including user name, post content, check-in representation, and image information;

[0051] The feature fusion module is used to combine the attributes of user name, post content, check-in representation and image information into a feature matrix, and construct a neural network structure diagram with users as nodes, relationships between users as edges, and feature matrices as nodes;

[0052] The node feature supplementation module is used to filter neighbor nodes after the graph neural network structure diagram, screen out nodes with small out-degree and in-degree, and retain those with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplement for the target node, and these supplementary information are weighted and integrated through the multi-head attention mechanism to further enhance the feature representation and information richness of the target node;

[0053] The identity association module is used to embed the identities of different users on the two platforms into the MLP classifier and output through softmax to determine whether the users on the two different platforms belong to the same user.

[0054] In a third aspect, the present invention provides an electronic device, comprising:

[0055] at least one processor; and,

[0056] a memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method for associating virtual identities based on a graph convolutional neural network with a large model and feature enhancement.

[0058] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the method for associating virtual identities using a graph convolutional neural network based on a large model and feature enhancement.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] The present invention utilizes the multi-level attributes of users to construct a virtual identity association model, including multiple processes such as user feature extraction, feature enhancement, network structure diagram, and large model fine-tuning. The multi-dimensional feature fusion method of the present invention can achieve high-precision user matching under weak supervision conditions, which is often not achieved in the prior art. In addition, the present invention utilizes a graph convolutional network (GCN) to process complex social network data, and optimizes the extraction and processing of information through a multi-head attention mechanism. This method can effectively process heterogeneous and multimodal data and enhance the learning and representation capabilities of the model. In the network structure diagram, an inhibitory multi-head attention mechanism is utilized to give priority to nodes with high topological importance. This design can improve the accuracy and efficiency of association analysis, and can greatly alleviate the imbalance of user information resources on different platforms, resulting in a low association success rate. And the use of fine-tuning large models can better achieve information association complementarity of different modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 This is a flowchart of a method for implementing virtual identity association based on a large model and feature-enhanced graph convolutional neural network according to an embodiment of the present invention;

[0063] Figure 2 This is a block diagram of an embodiment of the present invention that implements a virtual identity association system based on a large model and a feature-enhanced graph convolutional neural network.

[0064] Figure 3 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0066] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0067] like Figure 1 As shown, the present invention provides a method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement, comprising the following steps:

[0068] S1. Extract multiple attributes of the user that are most distinctive, including user name, post content, check-in representation, and image information.

[0069] S1.1, Username extraction;

[0070] Usernames are the data that best represents user characteristics. Extensive research has shown that users tend to use the same or similar usernames across different platforms. Because usernames can contain a wide variety of characters, such as letters, numbers, and special symbols, this example uses character-level attribute embedding to ensure textual similarity between similar users in order to fully extract username features.

[0071] Step S1.1 is specifically as follows:

[0072] S1.1.1. In this embodiment, the user name can be divided into unique character sequences. For example, "abandon_" can be divided into w = 'a': 2, 'b': 1, 'c': 0, 'd': 1, ..., 'n': 2, ..., 'o': 1, ..., others: 1. The numerical value represents the character count of the corresponding character in the text, and others represents all punctuation marks except numbers and English characters. Therefore, the count weight vector of the user name can be expressed as c i=[2,1,0,1…,2…,1,…1]; In order to reduce the impact of excessively large dimensions in deep learning, this embodiment uses a single-layer autoencoder to embed the count weighted vector into the distributed representation.

[0073] S1.1.2. This embodiment defines an autoencoder that maps an input vector x to a hidden representation z through a deterministic function z = f(x) = Wx + b. Through a decoding function y = g(z) = W * z+b * Restore the hidden feature representation z; the formula is as follows:

[0074] z=f(x)=Wx+b,

[0075] y=g(z)=W * z+b * ,

[0076] in, and are the weight matrix and bias vector respectively. is the weight matrix of the decoding function g(·).

[0077] The loss function of the autoencoder is defined as:

[0078] L=∑∥y i -c i ∥ 2

[0079] The encoder parameters W are trained by minimizing the loss function e ,b e , thus obtaining the user name feature representation X after dimensionality reduction c :

[0080] X ↓ =W e c+b e .

[0081] S1.2, post content extraction;

[0082] This embodiment innovatively uses a large language model (LLM) fine-tuning and customized prompt design method to extract user post features, performing a multimodal comprehensive analysis of post features. Through customized prompt templates, this embodiment can efficiently generate text descriptions and image features of posts. The specific design is as follows:

[0083] S1.2.1. Use multimodal image-text datasets to fine-tune large models to improve the comprehensive representation capabilities of the two modalities.

[0084] S1.2.2 Prepare a dataset of text, images, and their comprehensive representations, and construct the following data format:

[0085]

[0086] S1.2.3. Use instruction fine-tuning to improve the large model's ability to fine-tune specialized domains, thereby enhancing its multimodal comprehensive representation capabilities.

[0087] S1.2.4. Set the Prompt template to instruct the model to generate a text description of the post; the Prompt template is as follows: Prompt: {

[0088] ' <instuction>Generate a text description within 100characters of thetext and image of the social media post,describing it from a first-personperspective<\instruction>\n′

[0089] ′ <text>{{Text}}<\Text>\n′

[0090] ' {{Corr_image}}\n′}

[0091] S1.2.5, LLM C Represents the output of the large model; embed the user post text Corr_Text and the corresponding image Corr_image of the post into prompt. Through this structured instruction, we can ensure that the large language model accurately extracts the text and image features of the post, as follows:

[0092] Z=LLM C (Prompt,corr_Text,corr_image)

[0093] Among them, Z is generated by the Large Language Model (LLM), which combines the multimodal information of the post text and image and represents the overall characteristics of the post. Specifically, Z contains the semantic information extracted from the text description and the visual features extracted from the image. These features are processed and fused by the large language model to generate Z, which is a comprehensive description of the post.

[0094] S1.2.6. Further process Z using the BERT model to extract its deep text features. BERT (Bidirectional Encoder Representation Transformation), as a pre-trained language model, can capture rich semantic and contextual information in text and generate high-quality feature representations. The extraction process is as follows:

[0095] X w =BERT(Z)

[0096] S1.3. Check-in indication extraction:

[0097] Check-in posts reflect a user's daily life and are crucial for associating users with their identities. To represent user check-ins, a common approach is one-hot encoding, which uses a check-in attribute for each user's check-in location. However, this approach ignores the importance of the time dimension. Even if two users check in at the same location, they may be misclassified as the same user if their check-in times are different.

[0098] Therefore, this embodiment chooses to construct a spatiotemporal feature matrix to realize the modeling of user check-in information based on time period. Specifically, this embodiment divides time by month and sets the dimension D T Sequence. The locations are treated differently by country. Since the virtual identity is associated with a domestic account, the check-in points in China are divided by city, and the check-in points outside China are divided by country. For example, if user B has checked in in Beijing and Bangkok, Thailand, it will only be marked as having checked in in Beijing, China and Thailand. The final dimension is set as D L Sequence listing.

[0099] Because the dimension of the spatiotemporal feature matrix E is D L ×D T , at position (m,n) in the feature matrix, if the value is 1, it means that the user has visited location n within time period m. Accordingly, based on the check-in location and check-in time of the check-in post, the corresponding position in the feature matrix E is set to 1, indicating the user's check-in.

[0100] For users who haven't signed in to post, this invention uses IP address collection to fill in missing information. Some social platforms require IP address updates, so IP address collection can be performed for these platforms. For the collected time period and address, the corresponding position in the space-time matrix E is set to 1 for sign-in.

[0101] Because the address and time period sequences are large, the feature dimension of the check-in matrix is ​​too large, so we use the ST-GCN to reduce the dimension. Numerous experiments and papers have demonstrated that the spatiotemporal convolutional network (ST-GCN) is a neural network model that can efficiently process data with spatiotemporal structures. Therefore, we apply the ST-GCN to the feature matrix E. The resulting check-in representation can be obtained using the following formula:

[0102]

[0103] E=(T∪L ip )

[0104] S1.4. Image information extraction: For images posted by users, two solutions are adopted for parallel processing:

[0105] S1.4.1. RESNET performs image feature extraction; the details are as follows:

[0106] Users are accustomed to posting their favorite images on different social media platforms, which helps to achieve virtual identity association across different platforms. For users' visual posts, a residual network (ResNet) is used for encoding. Through this architecture, users' visual posts can be converted into a set of complex feature vectors. These vectors not only contain the original information of the image, but also contain deep semantic and style features.

[0107] To further refine and optimize the user's visual representation, this embodiment designs a unique mathematical model that combines pooling techniques, nonlinear activation functions, and fully connected network layers. This model not only processes high-dimensional data but also achieves dimensionality reduction and abstraction of features while maintaining information integrity.

[0108] This embodiment defines the following mathematical formula to formalize this process:

[0109] V=f avg (R(V1),R(V2),...,R(V n ))

[0110] V=g(W·f avg (V)+b)

[0111] Among them, V represents the optimized user visual feature vector, V i represents the feature encoding of the i-th visual post, R represents the encoding function of the ResNeT network, and f avg represents an advanced average pooling function for integrating the features of multiple visual posts, g is a nonlinear activation function used to introduce nonlinear characteristics and enhance the expressiveness of the model, W and b represent the weight matrix and bias vector of the fully connected layer, respectively.

[0112] S1.4.2, check-in attraction database identification;

[0113] Check-in attraction database identification: Users are accustomed to checking in at local attractions and uploading them to social networks. This paper uses the state-of-the-art and fastest YOLO V9 algorithm to detect objects in images and matches them with the local OpenImages Dataset, a large dataset containing millions of images, many of which are geotagged. This dataset also contains rich label information, making it suitable for this vision task.

[0114] Use the YOLOv9 algorithm to detect objects in user-uploaded images; let the input image be I, and the object detection algorithm output a set of bounding boxes and category labels Among them B i represents the bounding box of the i-th target, C i Represents its category label.

[0115]

[0116] Among them, Θ YOLOv9 are the parameters of the YOLOv9 model.

[0117] Match the detected targets with the images in the local database "Open Images Dataset"; let the images in the database and their geotags and label information be Among them G j Represents image I j Geographical tag, C j Indicates its label information.

[0118] The matching process can be defined as finding the database item that is most similar to the detected object in geotag and tag space:

[0119]

[0120] Among them, Sim(·,·) is a similarity metric function used to measure the similarity between the detected target and the database item. argmax() is used to extract the matching item with the highest similarity.

[0121] Finally, extract the geographic tags G of all users j , if the user image contains Beijing, China, Shenzhen, China, Bangkok, Thailand, and Seoul, South Korea, then construct the user image list U i = [Beijing, China, Shenzhen, China, Bangkok, Thailand, Seoul, South Korea]. Next, we use the pre-trained BERT model to encode the geographic tags and convert them into embedding vectors. The specific process is as follows:

[0122] Use the pre-trained BERT model to encode the text sequence of each geotag and generate the corresponding embedding vector. Let the geotag list be U i =[u1,u2,…,u n ], through the encoding process of the BERT model, the embedding vector E of each geographic tag can be obtained i =[e1,e2,…,e n ], where each e k Indicates geographic tag u k high-dimensional embedding vector of .

[0123] For each geotag u k , its embedding vector e k Obtained through the encoding process of the BERT model

[0124] e k =BERT(u k |Θ BERT )

[0125] Among them, Θ BERT Represents the pre-trained parameters of the BERT model.

[0126] S2. The attributes of user name, post content, check-in representation and image information are spliced ​​into a feature matrix, and a neural network structure diagram is constructed with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes.

[0127] After S3 and the graph neural network structure diagram are generated, neighbor nodes are screened, removing nodes with low out-degree and in-degree. This method retains only nodes with significant influence in the network, thereby enhancing the model's ability to capture important node relationships. At the same time, the features of the retained neighbor nodes are used as supplementary information for the target node. This supplementary information is weighted and integrated through a multi-head attention mechanism to further enhance the feature representation and information richness of the target node.

[0128] After obtaining the heterogeneous multimodal representation and social connection representation of each user, we should continue to study how to seamlessly integrate these representations to achieve accurate user representation learning. Since the user's social topology structure has been obtained, it is beneficial for the present invention to use neighbor node information to supplement user information.

[0129] This paper uses a graph convolutional network (GCN) to solve the problem of linking virtual identities across social networks. GCN has significant advantages in processing complex network data and can effectively utilize users' connection patterns and interaction information on different social platforms, thereby improving the accuracy and robustness of identity association. GCN encodes and concatenates the features of each user node (such as username, IP address, posted text, etc.), converting these features into vector representations in a high-dimensional space. This process uses the multiplication of the adjacency matrix and the feature matrix to extract deep features through a nonlinear transformation layer.

[0130] The user name, post, check-in and image attributes are spliced ​​into a feature matrix E, with users as nodes and relationships between users as edges. The social connections of user a are n represents that user a has n social connections. Since not every social connection has the same importance, the present invention defines the topological importance score s of neighbor nodes. i , where i represents the node index. Degree Centrality or other graph centrality metrics are used to calculate s i ,Right now:

[0131]

[0132] Where ∈ is a small constant used to prevent division by zero errors. degree(i) represents the degree of node i, that is, the number of nodes directly connected to node i.

[0133] At the same time, a multi-head attention layer is designed, and each head calculates the influence of neighboring nodes on the target node. The influence of node j on node i is calculated using the following formula:

[0134]

[0135] Where W (k) is the weight matrix of the head, a (k) is the attention vector, and ∥ represents vector connection.

[0136] According to s i The value of Adjustments are made to suppress nodes with low topological importance, which can be expressed as

[0137]

[0138] This step ensures that the model prioritizes nodes with high topological importance when aggregating neighbor node features.

[0139] Finally, the node features are updated using inhibitory multi-head attention:

[0140]

[0141] where σ is the nonlinear activation function and K is the number of attention heads.

[0142] S4. Embed the identities of different users on the two platforms into the MLP classifier and output them through softmax to determine whether the users on the two different platforms belong to the same user.

[0143] The previous module uses a graph neural network, and each user account corresponds to a node in the social network graph. Through the above method, each node i obtains a comprehensive feature representation h i For any two user nodes i and j, their feature vectors h i and h j Pairing is performed to form a joint feature vector. Common pairing methods include vector connection, vector difference or vector dot product, etc. The present invention adopts vector connection, that is:

[0144]

[0145] in Representation vector concatenation operation. This operation aims to combine the information in two feature vectors to construct a comprehensive joint feature representation.

[0146] Use a multilayer perceptron to process the joint feature vector h ij The MLP contains multiple hidden layers and nonlinear activation functions to enhance the model's ability to handle nonlinear data, such as ReLU, and finally performs classification through an output layer:

[0147] z ij =σ(W2·ReLU(W1h ij +b1)+b2)

[0148] Where W1, W2 and b1, b2 are the weight matrix and bias vector of the MLP, and σ is the sigmoid activation function of the output layer, which is used to normalize the output to between 0 and 1, that is, to output the probability that two nodes belong to the same natural person.

[0149] The probability value z based on the MLP output ij The value of is used to judge whether two users belong to the same natural person. Usually, by setting a threshold θ, if z ij ≥θ, then the two users are judged to be the same person.

[0150]

[0151] ·When z ij When ≥θ, the function outputs 1, indicating that according to the model, users i and j are the same natural person.

[0152] ·When z ij When <θ, the function outputs 0, indicating that users i and j are not the same natural person.

[0153] This paper proposes a method for building a virtual identity association model using multi-layered user attributes. This method can achieve highly accurate user matching in a weakly supervised environment. By integrating features from multiple dimensions, the present invention enhances the accuracy of user identity recognition.

[0154] Specifically, this technology uses graph convolutional networks (GCNs) to process complex social network data. GCNs can effectively handle heterogeneous and multimodal data, thereby enhancing the model's learning and representation capabilities. Furthermore, by introducing a multi-head attention mechanism, this invention further optimizes the information extraction and processing process.

[0155] At the network structure level, the present invention utilizes a multi-head attention mechanism to identify and prioritize nodes with higher topological importance in social networks, which helps to improve the accuracy and processing efficiency of association analysis.

[0156] In short, this paper provides an efficient and accurate method to process complex data sets and achieve effective matching of user identities by combining graph convolutional networks with a multi-head attention mechanism.

[0157] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0158] Based on the same idea as the method for implementing virtual identity association based on a large model and feature-enhanced graph convolutional neural network in the above-mentioned embodiment, the present invention also provides a system for implementing virtual identity association based on a large model and feature-enhanced graph convolutional neural network, which can be used to execute the above-mentioned method for implementing virtual identity association based on a large model and feature-enhanced graph convolutional neural network. For ease of explanation, the structural diagram of the embodiment of the system for implementing virtual identity association based on a large model and feature-enhanced graph convolutional neural network only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or a combination of certain components, or a different arrangement of components.

[0159] See also Figure 2 In another embodiment of the present application, a virtual identity association system 100 is provided based on a large model and feature enhanced graph convolutional neural network. The system includes a feature extraction module 101, a feature fusion module 102, a node feature supplementation module 103, and an identity association module 104.

[0160] The feature extraction module 101 is used to extract multiple attributes of the user that are most distinctive, including user name, post content, check-in information, and image information;

[0161] The feature fusion module 102 is used to splice the attributes of user name, post content, check-in representation and image information into a feature matrix, and construct a graph neural network structure diagram with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes; the node feature supplement module 103 is used to filter neighbor nodes after the graph neural network structure diagram, screen out nodes with small out-degree and in-degree, and retain those nodes with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplements for the target node, and these supplementary information are weighted and integrated through a multi-head attention mechanism to further enhance the feature representation and information richness of the target node.

[0162] The identity association module 104 is used to embed the identities of different users on the two platforms into the MLP classifier and output through softmax to determine whether the users on the two different platforms are the same user.

[0163] It should be noted that the virtual identity association system based on a large model and feature-enhanced graph convolutional neural network of the present invention corresponds one-to-one to the virtual identity association method based on a large model and feature-enhanced graph convolutional neural network of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the virtual identity association method based on a large model and feature-enhanced graph convolutional neural network are applicable to the embodiment of the virtual identity association based on a large model and feature-enhanced graph convolutional neural network. For specific details, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0164] In addition, in the implementation of the virtual identity association system based on the large model and feature enhanced graph convolutional neural network in the above embodiment, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the convenience of software implementation, such as the configuration requirements of the corresponding hardware. That is, the internal structure of the virtual identity association system based on the large model and feature enhanced graph convolutional neural network is divided into different program modules to complete all or part of the functions described above.

[0165] See also Figure 3 In one embodiment, an electronic device is provided for implementing a method for associating a virtual identity using a graph convolutional neural network with a large model and feature enhancement. The electronic device 200 may include a first processor 201, a first memory 202, and a bus. It may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a program 203 for associating a virtual identity using a graph convolutional neural network with a large model and feature enhancement.

[0166] The first memory 202 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, smart memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed on the electronic device 200, such as the code of the virtual identity association program 203 based on a large model and feature enhancement graph convolutional neural network, but can also be used to temporarily store data that has been output or is about to be output.

[0167] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0168] Figure 3 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0169] The virtual identity association program 203 based on a large model and feature-enhanced graph convolutional neural network stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0170] Extracting multiple attributes of the user that are most distinctive, including username, post content, check-in information, and image information;

[0171] The attributes of user names, post content, check-in representations, and image information are concatenated into a feature matrix. A graph neural network structure is constructed with users as nodes, relationships between users as edges, and the feature matrix as node attributes. After constructing the graph neural network structure, neighbor nodes are filtered, removing nodes with low out-degree and in-degree. This method retains only nodes with significant influence in the network, enhancing the model's ability to capture important node relationships. Furthermore, the features of the retained neighbor nodes are used as supplementary information for the target node. This supplementary information is weighted and integrated through a multi-head attention mechanism to further enhance the feature representation and information richness of the target node.

[0172] The identities of different users on the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users on two different platforms belong to the same user.

[0173] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0174] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0175] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.< / text> < / instuction>

Claims

1. A virtual identity association method based on a large model and feature-enhanced graph convolutional neural network, characterized by: The steps include: Fine-tune the multimodal large model and extract the most discriminative attributes of users, including username, post content, check-in information, and image information; The attributes of user name, post content, check-in representation, and image information are spliced ​​into a feature matrix, and a neural network structure diagram is constructed with users as nodes, relationships between users as edges, and feature matrices as attributes of nodes; After constructing the graph neural network structure, the neighbor nodes are screened, and nodes with small out-degree and in-degree are removed to retain those with significant influence in the network, thereby enhancing the model's ability to capture important node relationships. The features of the retained neighbor nodes are used as information supplementary to the target node, and this supplementary information is weighted and integrated through a multi-head attention mechanism to further enhance the feature representation and information richness of the target node. The identities of different users on the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users on two different platforms belong to the same user; The feature representation of the enhanced target node is specifically: User a's social connections are n represents that user a has n social connections. Since not every social connection has the same importance, the topological importance score s of the neighbor nodes is defined as i , where i represents the node index, and degree centrality or other graph centrality indicators are used to calculate s i ,Right now: Where ∈ is a small constant used to prevent division by zero errors, and degree(i) represents the degree of node i, that is, the number of nodes directly connected to node i; Design a multi-head attention layer, where each head calculates the influence of neighboring nodes on the target node. The influence of node j on node i is calculated using the following formula: Where W (k) is the weight matrix of the head, a (k) is the attention vector, || represents vector connection, h i represents the feature vector of node i, h j represents the feature vector of node j; According to s i The value of Adjustments are made to suppress nodes with low topological importance, specifically expressed as follows: Updating node features using inhibitory multi-head attention.

2. The method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement according to claim 1 is characterized in that: The method for extracting the username is as follows: Divide the user name into a unique character sequence based on the characters, wherein the character sequence includes characters and numerical values, and the numerical values ​​represent the character count of the corresponding characters in the user name; Get the count weighted vector of the username based on the character sequence; The feature vector representation of the final user name is obtained based on the preset single-layer autoencoder.

3. The method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement according to claim 1 is characterized in that: The method for extracting the post content is as follows: Embed the user post text corr_Text and the corresponding image corr_image into the preset Prompt template; The above prompt template is used as the input of the pre-trained large language model, so that the large language model can accurately extract the text features and image features Z of the post; Z=LLM C (Prompt,corr_Text,corr_image); Among them, Z is composed of the large language model LLM C The output combines multimodal information from the post text and the corresponding image, representing the overall characteristics of the post. Z contains semantic information extracted from the text description and visual features extracted from the image. These features are processed and integrated by a large language model to generate Z, which is a comprehensive description of the post. Z is further processed using the pre-trained BERT model to extract deep text features.

4. The method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement according to claim 1 is characterized in that: The extraction of the sign-in representation is specifically as follows: Select and construct a spatiotemporal feature matrix to realize the modeling of user check-in information based on time period. The spatiotemporal feature matrix dimension is D L ×D T , at the (m,n) position of the spatiotemporal feature matrix, if the value is 1, it means that the user has arrived at location n within time period m; accordingly, according to the check-in location and check-in time of the check-in post, the corresponding position in the spatiotemporal feature matrix is ​​set to 1, indicating the user's check-in; For users who have not posted a sign-in post, the IP address is used to fill in the missing information; the corresponding position of the spatiotemporal feature matrix of the collection time period and its address is set to 1 for sign-in; The spatiotemporal feature matrix is ​​processed using the spatiotemporal convolutional network ST-GCN to obtain the final sign-in representation.

5. The method for implementing virtual identity association based on a graph convolutional neural network with a large model and feature enhancement according to claim 1 is characterized in that: The image information extraction is specifically as follows: Extract image features based on residual network; Perform object detection on user-uploaded images based on the YOLOv9 algorithm and match the detected objects with images in the local database. The matching process is defined as finding the database item that is most similar to the detected object in the geotag and tag space. Extract the geotags of all users, encode them using the pre-trained BERT model, and convert them into embedding vectors.

6. The method for associating virtual identities using a graph convolutional neural network based on a large model and feature enhancement according to claim 1, characterized in that: The identities of different users on the two platforms are embedded into the MLP classifier and output through softmax to determine whether the users on the two different platforms belong to the same user. Specifically: For any two user nodes i and j, their feature vectors h i and h j Pair them to form a joint feature h ij ; Use a multi-layer perceptron (MLP) to process the joint feature vector h ij ; MLP contains multiple hidden layers and nonlinear activation functions to enhance the model's ability to handle nonlinear data, and finally performs classification through an output layer: With ij =σ(W2 ReLU(W1h ij +b1)+b2) Where W1, W2 and b1, b2 are the weight matrix and bias vector of the MLP, and σ is the sigmoid activation function of the output layer, which is used to normalize the output to between 0 and 1, that is, to output the probability that two nodes belong to the same natural person; The probability value z based on the MLP output ij The value of z determines whether two users belong to the same natural person; by setting a threshold θ, if z ij ≥θ, then the two users are judged to be the same person; When z ij When ≥θ, the function outputs 1, indicating that according to the model, users i and j are the same natural person; When z ij When <θ, the function outputs 0, indicating that users i and j are not the same natural person.

7. A virtual identity association system is implemented based on a large model and feature-enhanced graph convolutional neural network, which is characterized by: A method for implementing virtual identity association using a graph convolutional neural network based on a large model and feature enhancement, as described in any one of claims 1 to 6, comprising a feature extraction module, a feature fusion module, a node feature supplementation module, and an identity association module; The feature extraction module is used to extract multiple attributes of the user that are most distinctive, including user name, post content, check-in representation, and image information; The feature fusion module is used to combine the attributes of user name, post content, check-in representation and image information into a feature matrix, and construct a neural network structure diagram with users as nodes, relationships between users as edges, and feature matrices as nodes; The node feature supplementation module is used to screen neighbor nodes after the graph neural network structure diagram, screen out nodes with small out-degree and in-degree, and retain those nodes with significant influence in the network, thereby enhancing the model's ability to capture important node relationships; the features of the retained neighbor nodes are used as information supplements for the target node, and these supplementary information are weighted and integrated through the multi-head attention mechanism to further enhance the feature representation and information richness of the target node; the identity association module is used to embed the identities of different users on the two platforms into the MLP classifier and output through softmax to determine whether users on two different platforms belong to the same user.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method for implementing virtual identity association based on a large model and feature enhanced graph convolutional neural network as described in any one of claims 1-6.

9. A computer-readable storage medium storing a program, characterized in that: When the program is executed by the processor, the method for associating virtual identities based on a graph convolutional neural network with a large model and feature enhancement as described in any one of claims 1 to 6 is implemented.

Images