Cross-social network virtual identity association method and device based on multimodal fusion and representation alignment

Abstract

The present invention discloses a method and device for associating virtual identities across social networks based on multimodal fusion and representation alignment. The method comprises: extracting features from user names, user-posted text, and user social relationships on social networks of different platforms to obtain feature information of different user modalities; utilizing an attention mechanism to perform multimodal fusion based on the feature information to obtain a first user representation that integrates multidimensional features; strengthening the first user representation through representation alignment to ultimately obtain a second user representation with the same distribution across different platforms; calculating the cosine similarity between the second user representations to obtain a similarity score between users, and selecting the user pair with the highest score as the identity association result. The present invention addresses the problem of a single modality model being unable to fully describe users and the problem of distribution differences across different social platforms through the methods of multimodal fusion and representation alignment.

Description

Technical Field

[0001] The present invention belongs to the technical field of social network virtual identity association, and specifically relates to a method and device for cross-social network virtual identity association with multimodal fusion and representation alignment. Background Art

[0002] Nowadays, social networks, with their high convenience, have become an indispensable part of people's lives. People often join multiple social platforms to enjoy different services, such as using WeChat for communication and Weibo for news and check-ins. Consequently, numerous scholars have devoted themselves to research related to social networks. Cross-social network virtual identity association, a key component of this research, aims to identify the social accounts of the same individual across different platforms. This research has attracted significant attention in fields such as recommendation systems, user behavior analysis, and information dissemination.

[0003] In fact, numerous methods have been proposed for linking user identities. Current approaches can be broadly categorized into three main categories: those based on user attributes, those based on user-generated content, and those based on user social relationships. However, each of these approaches has drawbacks. For privacy reasons, users selectively disclose their profile attributes and keep sensitive information (such as age or contact information) confidential. They may even fabricate or imitate information, increasing uncertainty and ambiguity. Due to the richness of social networks, user posts can take on a variety of forms (text, images, etc.). Using only a single type of content can result in information loss. Existing methods based on user social relationships overemphasize structured information. However, the characteristics of a user's friends in social networks can also be very helpful in identifying users. After all, the characteristics of friends can sometimes be more unique than the user's own characteristics. Incorporating the characteristics of their friends significantly improves accuracy. Therefore, multimodal user information should be utilized, rather than limited to a single modality. Furthermore, the confidence with which a modality portrays a user varies. Sometimes a user's text conveys more information than other modalities, while sometimes images also play a key role. Therefore, adaptively characterizing different modes is the key to solving this problem.

[0004] Secondly, while the same user may post similar information on different social platforms, due to the inconsistent data distribution between platforms, this similar information may also have different representations. However, existing methods often directly link user identities based on their representations without considering the semantic differences between them. Therefore, how to make the representations of the same user on different platforms similar is another major challenge. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method and device for associating virtual identities across social networks based on multimodal fusion and representation alignment. The method of multimodal fusion and representation alignment solves the problem that a single model cannot fully describe users and that there are distribution differences on different social platforms.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for associating virtual identities across social networks based on multimodal fusion and representation alignment, comprising the following steps:

[0008] Extract features from the user names, texts posted by users, and social relationships of users on different social networks to obtain user name feature information, user text feature information, and user social relationship feature information respectively;

[0009] Based on the obtained username feature information, the text feature information posted by the user, and the user social relationship feature information, an attention mechanism is used to perform multimodal fusion to obtain a first user representation that integrates multi-dimensional features;

[0010] Performing user representation enhancement processing on the first user representation using a representation alignment method, and ultimately obtaining a second user representation having the same distribution space across different platforms;

[0011] The cosine similarity between the second user representations is calculated to obtain a similarity score between the users, and the user pair with the highest score is used as the identity association result.

[0012] As a preferred technical solution, the feature extraction of the user name is specifically as follows:

[0013] For a given user's username, we use the character-level Bag-of-Words model to extract features and count the number of times each character in each username appears to obtain a vector. All the obtained user name vectors are concatenated in sequence to obtain the user name count matrix Since C0 is a sparse matrix, an automatic encoder is used to convert it. The conversion formula is as follows:

[0014]

[0015] Among them, W e , b e are the weights and biases of the encoder, W d , b d are the decoder weights and biases, C 1 is the decoder username vector matrix, and They are user name vectors respectively, and the loss function L is continuously trained through gradient descent c , get the optimal W e and b e , and finally get the username embedding matrix with dimension d

[0016] As a preferred technical solution, the feature extraction of the text published by the user is specifically as follows:

[0017] The text published by the user is input into the Word2Vec model to obtain the embedding vector of each text. Then the embedding vector of each user's text is averaged as the representation of the user's text. The text embedding vectors of all users are spliced in sequence to obtain a text embedding matrix with a dimension of d.

[0018] As a preferred technical solution, the feature extraction of user social relationships is specifically as follows:

[0019] Will be provided by platform N 1 n users and platforms N 2 The n×m adjacency matrix of the social relationships of m users is obtained by DeepWalk model to obtain the embedding vector of each user's social relationship. The social relationship embedding vectors of all users are spliced in sequence to obtain the user social relationship embedding matrix with dimension d.

[0020] As a preferred technical solution, the multimodal fusion is to embed the three user feature information into a matrix, use the attention mechanism to perform multimodal fusion, give different weights to each modality to reflect the importance between different modalities, and after multimodal fusion, obtain the first user representation matrix Z f The calculation formula is:

[0021]

[0022] Among them, α c , α T , α V The weights of the user name, text, and social relationship embedding matrices respectively; f(.) is the attention network.

[0023] As a preferred technical solution, the specific steps of the representation alignment to enhance user representation are:

[0024] First, the first user representation is placed in a fully connected layer to map the user representations of the two platforms into the same space to obtain the second user representation. The calculation formula for the second user representation is:

[0025]

[0026] Among them, W l , b l are the weight and bias of the fully connected layer respectively, is the first user representation obtained by multimodal fusion of platform N, and Z is the second user representation;

[0027] Secondly, to train all weights and biases in this method, minimizing the EMD distance is used as the first optimization goal, and the calculation formula for the first optimization goal is:

[0028]

[0029]

[0030] Among them, L E is the first optimization objective, d ij For users The second user said and users The second user said The distance, F ij For users and users The probability of association between represents the square of the F norm;

[0031] In addition, by reducing the representation distance between user pairs and P ij and F ij The difference between them is used to set a second optimization objective to better guide the learning of the second user representation. The calculation formula of the second optimization objective is:

[0032]

[0033] Among them, L R is the second optimization objective, n p is the number of associated user sample pairs, λ1 and λ2 are hyperparameters, and for associated user sample pairs, the true association probability P ij =1;

[0034] The final optimization goal L is the sum of the first optimization goal and the second optimization goal, that is:

[0035] L=L E +L R

[0036] Finally, L is continuously optimized by gradient descent method to obtain the optimal weight and bias, and finally the optimal W is obtained. l and b l The second user representation Z is obtained.

[0037] As a preferred technical solution, the identity association result is obtained by calculating the cosine similarity between the second user representations, and the calculation formula is as follows:

[0038]

[0039] in, For platform N 1 Users The second user represents and For platform N 2 Users The second user said that S ij For users and users The cosine similarity of .

[0040] In a second aspect, the present invention further provides a cross-social network virtual identity association system based on multimodal fusion and representation alignment, which applies the cross-social network virtual identity association method based on multimodal fusion and representation alignment, including a feature extraction module, a multimodal fusion module, a representation alignment module, and an identity association module;

[0041] The feature extraction module is used to extract features from user names, texts posted by users, and social relationships of users on social networks of different platforms, and obtain user name feature information, user text feature information, and user social relationship feature information respectively;

[0042] The multimodal fusion module is used to perform multimodal fusion based on the three types of user feature information using an attention mechanism to obtain a first user representation that integrates multi-dimensional features;

[0043] The representation alignment module is configured to strengthen the first user representation by representation alignment, and ultimately obtain a second user representation with the same distribution across different platforms;

[0044] The identity association module is used to calculate the cosine similarity between the second user representations to obtain similarity scores between users, and use the user pair with the highest score as the identity association result.

[0045] In a third aspect, the present invention further provides an electronic device, comprising:

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the cross-social network virtual identity association method based on multimodal fusion and representation alignment.

[0049] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the cross-social network virtual identity association method based on multimodal fusion and representation alignment.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] 1. This invention fully exploits three modal information types: username, user content, and social relationships. It uses a character-level Bag-of-Words model to extract features specific to each username. It also extracts text and social relationship features from the Word2Vec and DeepWalk models. Based on this, it uses an attention mechanism to automatically learn feature weights for multimodal fusion. This addresses the issue of existing methods failing to fully describe users using a single modality or failing to perfectly fuse multimodal information.

[0052] 2. After obtaining the user representation, the present invention further strengthens the user representation through representation alignment, so that the user representations belonging to the same natural person on different platforms are as close as possible, solving the problem of data distribution differences on different social platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 This is a framework diagram of a method for associating virtual identities across social networks based on multimodal fusion and representation alignment according to an embodiment of the present invention;

[0055] Figure 2 This is a block diagram of a cross-social network virtual identity association system based on multimodal fusion and representation alignment according to an embodiment of the present invention;

[0056] Figure 3 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0059] Multimodal fusion: Multimodal fusion is the process of combining information from two or more modalities to make predictions. A single modality often cannot contain all the valid information required to produce accurate predictions. Multimodal fusion combines information from two or more modalities to complement this information, broaden the coverage of the input data, improve the accuracy of predictions, and enhance the robustness of the prediction model.

[0060] See also Figure 1 In one embodiment of the present application, a method for associating virtual identities across social networks with multimodal fusion and representation alignment is provided, comprising the following steps:

[0061] S1. Extract features of user names, user-posted texts, and user social relationships on different social networks to obtain user name feature information, user-posted text feature information, and user social relationship feature information, respectively.

[0062] S11, feature extraction of the user name, specifically:

[0063] For a given user’s username “abza12”, we use the character-level Bag-of-Words model to extract features and count the number of times each character in each username appears to obtain a vector. For example, in the username "abza12", a:2, b:1, z:1, "1":1, "2":1, so we get the vector All the obtained user name vectors are concatenated in sequence to obtain the user name count matrix Since C0 is a sparse matrix, an automatic encoder is used to convert it. The conversion formula is as follows:

[0064]

[0065] Among them, W e , b e are the weights and biases of the encoder, W d , b d are the decoder weights and biases, and They are user name vectors respectively, and L is trained continuously through gradient descent c , get the optimal W e and b e , and finally get the username embedding matrix with dimension d

[0066] S12, feature extraction of the text published by the user, specifically:

[0067] For user-posted text, such as "Today is a sunny day.", after removing stop words, the resulting text is input into the Word2Vec model to obtain the vector representation of each word. The word vectors of each text are further summed to obtain the embedding vector of each text. The embedding vectors of all texts posted by each user are averaged as the representation of the user's posted text. The text embedding vectors of all users are spliced in sequence to obtain a text embedding matrix with a dimension of d.

[0068] S13, feature extraction of the user social relationship, specifically:

[0069] For user social relationships, if platform N 1 The i-th user and platform N 2 If the jth user has a friend relationship, the ijth position of its adjacency matrix is set to 1, and the platform N 1 n users and platforms N 2 The n×m adjacency matrix of the social relationships of m users is obtained by DeepWalk model to obtain the embedding vector of each user's social relationship. The social relationship embedding vectors of all users are spliced in sequence to obtain the user social relationship embedding matrix with dimension d.

[0070] S2. Based on the feature information, use the attention mechanism to perform multimodal fusion to obtain a first user representation that integrates multi-dimensional features.

[0071] S21, the multimodal fusion is to embed the three user feature information into a matrix; use the attention mechanism to perform multimodal fusion, give different weights to each modality to reflect the importance between different modalities, and after multimodal fusion, obtain the first user representation matrix Z f The calculation formula is:

[0072]

[0073] Among them, α C , α T , α V The weights of the user name, text, and social relationship embedding matrices respectively; f(.) is the attention network.

[0074] S3. Strengthen the first user representation by aligning the representations, and finally obtain second user representations with the same distribution on different platforms.

[0075] S31. First, the first user representation is placed in a fully connected layer to map the user representations of the two platforms into the same space to obtain a second user representation. The calculation formula for the second user representation is:

[0076]

[0077] Among them, W l , b l are the weight and bias of the fully connected layer respectively, is the first user representation obtained by multimodal fusion of platform N, and Z is the second user representation.

[0078] S32. Secondly, minimizing the EMD distance is used as the first optimization goal. The calculation formula of the first optimization goal is:

[0079]

[0080]

[0081] Among them, L E is the first optimization objective, d ij For users The second user said and users The second user said The distance, F ij For users and users The probability of association between represents the square of the F-norm.

[0082] S33, this method reduces the distance between the user pairs and P ij and F ij The difference between them is used to set a second optimization objective to better guide the learning of the second user representation. The calculation formula of the second optimization objective is:

[0083]

[0084] Among them, LR is the second optimization objective, n p is the number of associated user sample pairs, λ1 and λ2 are hyperparameters, and for associated user sample pairs, the true association probability P ij =1;

[0085] The final optimization goal L is the sum of the first optimization goal and the second optimization goal, that is:

[0086] L=L E +L R

[0087] Finally, L is continuously optimized by gradient descent method to obtain the optimal weight and bias, and finally the optimal W is obtained. l and b l The second user representation Z is obtained.

[0088] S4. Calculate the cosine similarity between the second user representations to obtain a similarity score between the users, and use the user pair with the highest score as the identity association result.

[0089] S41, computing platform N 1 With platform N 2 Second user representation cosine similarity between users:

[0090]

[0091] in, For platform N 1 Users The second user represents and For platform N 2 Users The second user said that S ij For users and users Finally, the similarity between users is scored, and the user pair with the highest score is taken as the identity association result.

[0092] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0093] Based on the same concept as the cross-social network virtual identity association method based on multimodal fusion and representation alignment in the above-mentioned embodiment, the present invention also provides a cross-social network virtual identity association system based on multimodal fusion and representation alignment, which can be used to implement the above-mentioned cross-social network virtual identity association method based on multimodal fusion and representation alignment. For ease of illustration, the structural diagram of the embodiment of the cross-social network virtual identity association system based on multimodal fusion and representation alignment only shows the parts relevant to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0094] See also Figure 2 In another embodiment of the present application, a cross-social network virtual identity association system 100 based on multimodal fusion and representation alignment is provided. The system includes a feature extraction module 101, a multimodal fusion module 102, a representation alignment module 103, and an identity association module 104.

[0095] The feature extraction module 101 is used to extract features from user names, user-posted texts, and user social relationships on social networks of different platforms, and obtain user name feature information, user-posted text feature information, and user social relationship feature information respectively;

[0096] The multimodal fusion module 102 is configured to perform multimodal fusion based on the three user feature information using an attention mechanism to obtain a first user representation that integrates multi-dimensional features;

[0097] The representation alignment module 103 is configured to enhance the first user representation by representation alignment, and ultimately obtain a second user representation with the same distribution across different platforms;

[0098] The identity association module 104 is configured to calculate the cosine similarity between the second user representations to obtain similarity scores between the users, and use the user pair with the highest score as the identity association result.

[0099] It should be noted that the cross-social network virtual identity association system based on multimodal fusion and representation alignment of the present invention corresponds one-to-one to the cross-social network virtual identity association method based on multimodal fusion and representation alignment of the present invention. The technical features and beneficial effects described in the embodiment of the cross-social network virtual identity association method based on multimodal fusion and representation alignment are all applicable to the embodiment of the cross-social network virtual identity association system based on multimodal fusion and representation alignment. For specific details, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0100] In addition, in the implementation of the cross-social network virtual identity association system based on multimodal fusion and representation alignment in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the cross-social network virtual identity association system based on multimodal fusion and representation alignment is divided into different program modules to complete all or part of the functions described above.

[0101] See also Figure 3 In one embodiment, an electronic device is provided for implementing a method for associating virtual identities across social networks based on multimodal fusion and representation alignment. The electronic device 200 may include a first processor 201, a first memory 202, and a bus. It may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a program 203 for associating virtual identities across social networks based on multimodal fusion and representation alignment.

[0102] The first memory 202 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 may include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed on the electronic device 200, such as the code of the cross-social network virtual identity association program 203 for multimodal fusion and representation alignment, but can also be used to temporarily store data that has been output or is about to be output.

[0103] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0104] Figure 3 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0105] The multimodal fusion and representation alignment cross-social network virtual identity association program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When executed in the first processor 201, it can achieve:

[0106] Perform feature extraction on the user names, texts posted by users, and social relationships of users on different social networks to obtain user name feature information, user text feature information, and user social relationship feature information respectively;

[0107] Based on the feature information, an attention mechanism is used to perform multimodal fusion to obtain a first user representation that integrates multi-dimensional features;

[0108] Strengthening the first user representations by representation alignment, and ultimately obtaining second user representations with the same distribution across different platforms;

[0109] The cosine similarity between the second user representations is calculated to obtain a similarity score between the users, and the user pair with the highest score is used as the identity association result.

[0110] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0111] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A cross-social network virtual identity association method based on multimodal fusion and representation alignment, characterized by: The steps include: Extract features from the user names, texts posted by users, and social relationships of users on different social networks to obtain user name feature information, user text feature information, and user social relationship feature information respectively; Based on the obtained username feature information, user-posted text feature information, and user social relationship feature information, the attention mechanism is used for multimodal fusion to obtain a first user representation that integrates multi-dimensional features; The first user representation is enhanced by a representation alignment method, and finally a second user representation having the same distribution space on different platforms is obtained. The specific steps of enhancing the user representation by representation alignment are: First, the first user representation is placed in a fully connected layer to map the user representations of the two platforms into the same space to obtain the second user representation. The calculation formula for the second user representation is: Among them, W l ,b l are the weight and bias of the fully connected layer respectively, is the first user representation obtained by multimodal fusion of platform N, and Z is the second user representation; Secondly, to train all weights and biases in this method, minimizing the EMD distance is used as the first optimization goal, and the calculation formula for the first optimization goal is: Among them, L E is the first optimization objective, d ij For users The second user said and users The second user said The distance, F ij For users and users The probability of association between represents the square of the F norm; In addition, by reducing the representation distance between user pairs and P ij and F ij The difference between them is used to set a second optimization objective to better guide the learning of the second user representation. The calculation formula of the second optimization objective is: Among them, L R is the second optimization objective, n p is the number of associated user sample pairs, λ1 and λ2 are hyperparameters, and for associated user sample pairs, the true association probability P ij =1; The final optimization goal L is the sum of the first optimization goal and the second optimization goal, that is: L=L E +L R Finally, L is continuously optimized by gradient descent method to obtain the optimal weight and bias, and finally the optimal W is obtained. l and b l Obtain a second user representation Z; The cosine similarity between the second user representations is calculated to obtain a similarity score between the users, and the user pair with the highest score is used as the identity association result.

2. The cross-social network virtual identity association method based on multimodal fusion and representation alignment according to claim 1 is characterized in that: The feature extraction of the user name is specifically as follows: For a given user's username, we use the character-level Bag-of-Words model to extract features and count the number of times each character in each username appears to obtain a vector. All the obtained user name vectors are concatenated in sequence to obtain the user name count matrix Because C 0 It is a sparse matrix, so an automatic encoder is used to convert it. The conversion formula is as follows: Among them, W e ,b e are the weights and biases of the encoder, W d ,b d are the decoder weights and biases, C 1 is the decoder username vector matrix, and They are user name vectors respectively, and the loss function L is continuously trained through gradient descent c , get the optimal W e and b e , and finally get the username embedding matrix with dimension d 3. The cross-social network virtual identity association method based on multimodal fusion and representation alignment according to claim 1 is characterized in that: The feature extraction of the text published by the user is specifically as follows: The text published by the user is input into the Word2Vec model to obtain the embedding vector of each text. Then the embedding vector of each user's text is averaged as the representation of the user's text. The text embedding vectors of all users are spliced in sequence to obtain a text embedding matrix with a dimension of d.

4. The method for associating virtual identities across social networks based on multimodal fusion and representation alignment according to claim 1 is characterized in that: The feature extraction of the user social relationship is specifically as follows: Will be provided by platform N 1 n users and platforms N 2 The n×m adjacency matrix of the social relationships of m users is obtained by DeepWalk model to obtain the embedding vector of each user's social relationship. The social relationship embedding vectors of all users are spliced in sequence to obtain the user social relationship embedding matrix with dimension d 5. The method for associating virtual identities across social networks based on multimodal fusion and representation alignment according to claim 1, characterized in that: The multimodal fusion is to embed the three user feature information into a matrix, and use the attention mechanism to perform multimodal fusion, giving different weights to each modality to reflect the importance between different modalities. After multimodal fusion, the first user representation matrix Z is obtained. f The calculation formula is: Among them, α C ,α T ,α V The weights of the user name, text, and social relationship embedding matrices respectively; f(.) is the attention network.

6. The method for associating virtual identities across social networks based on multimodal fusion and representation alignment according to claim 1, characterized in that: The identity association result is obtained by calculating the cosine similarity between the second user representations, and the calculation formula is as follows: in, For platform N 1 Users The second user represents and For platform N 2 Users The second user said that S ij For users and users The cosine similarity of .

7. A cross-social network virtual identity association system based on multimodal fusion and representation alignment, characterized by: A cross-social network virtual identity association method for multimodal fusion and representation alignment applied to any one of claims 1-6, comprising a feature extraction module, a multimodal fusion module, a representation alignment module, and an identity association module; The feature extraction module is used to extract features from user names, texts posted by users, and social relationships of users on social networks of different platforms, and obtain user name feature information, user text feature information, and user social relationship feature information respectively; The multimodal fusion module is used to perform multimodal fusion based on the three user feature information using the attention mechanism to obtain a first user representation that integrates multi-dimensional features; The representation alignment module is configured to strengthen the first user representation by representation alignment, and ultimately obtain a second user representation with the same distribution across different platforms; The identity association module is used to calculate the cosine similarity between the second user representations to obtain similarity scores between users, and use the user pair with the highest score as the identity association result.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the cross-social network virtual identity association method based on multimodal fusion and representation alignment as described in any one of claims 1-6.

9. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the method for associating virtual identities across social networks based on multimodal fusion and representation alignment according to any one of claims 1 to 6 is implemented.

Images