A case big data network identity association and character portrait method

Abstract

The application provides a case big data network identity correlation and character portrait method, the method comprises the following steps: step 1, obtaining case data; step 2, the data obtained is cleaned; step 3, important targets are identified by using overlapping community discovery technology; step 4, user blog vectorization; step 5, user personal information vectorization; step 6, blog similarity calculation of source user and candidate user.The application can process the case data well; in the application, the important target identification technology based on overlapping community discovery is used, the local clustering structure is used as a guide to detect overlapping communities, and the complex overlapping clustering problem is reduced to a non-overlapping partition problem which is simpler and easier to handle; the application uses vector representation, behavior correlation and other methods to construct a network user identity correlation model based on multidimensional features, and mines cross-social platform user correlation and restores user social relationship network.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, in particular, a case big data network identity association and character portrait method. BACKGROUND

[0002] Network identity association and character portrait can assist relevant departments to more quickly and effectively identify personnel information and the relationship between personnel and other personnel, and provide rich data support for risk early warning and other work. However, the current processing method has the problems of trivial personnel identity association and low efficiency of character relationship extraction in the process of personnel network identity association.

[0003] There is a lack of clear macro community structure in the network of the real world, that is, it is difficult to observe a community with a clear clustering attribute, and it is difficult to detect using common algorithms. The reason is that the clusters in the real network overlap each other, and many edges cross the cluster boundary. Therefore, the network graph of the real world does not show a clear cluster structure at the macro level. However, although the community detection problem is difficult at the macro level, it is relatively simple at the micro level. Because even if a node is part of many communities, if we limit a node and one of its neighbors, there will be only one or a limited number of communities between the two nodes that interact.

[0004] In addition, due to the multi-source heterogeneity of user data in network activities, the ideas for research based on different data types are very different. The current main research routes are: identity association based on user attributes, that is, similarity calculation is performed on user attribute data common to different platforms, and then classification or weighting methods are used to determine whether two users belong to the same natural person; identity association based on user relationship, one is a prior node method, and the other is a non-prior node method; identity association based on user-generated content, there are many related researches in the field of natural language processing. However, in the task of cross-social network user identity association, the text object is mostly non-standard short text, and the user original content is less, so it is difficult to associate only through the text content. Therefore, under the condition of data support, the identity association is carried out by combining the time, space and other information attached to the user-generated content.

[0005] The current technical research on user identity association is to divide and cluster users through data mining or machine learning methods, so as to associate the identity. The main research ideas are divided into virtual identity association technology based on user registration information, virtual identity association technology based on user-generated content, and virtual identity association technology based on user social relationship in social network platform. However, these technologies still have the problems of low recall rate, low accuracy, or low efficiency. SUMMARY

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for linking network identities and creating profiles based on big data in cases.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] This invention provides a method for linking network identities and creating user profiles based on big data in case studies. The method includes:

[0009] Step 1: Obtain case data;

[0010] Step 2: Clean the acquired data;

[0011] Step 3: Identify important targets using overlapping community detection technology;

[0012] Step 4: Vectorize user blog posts;

[0013] Step 5: Vectorize user personal information;

[0014] Step 6: Calculate the similarity between the blog posts of the source user and the candidate user.

[0015] Based on the above, the case data in step 1 includes: user ID, user nickname, IP address, superior, and real name.

[0016] Based on the above, the data cleaning in step 2 includes:

[0017] Step 2-1: If there are blank entries in the data, delete that entry; if there are multiple records with completely duplicate attribute items, keep only one record and delete the rest.

[0018] Step 2-2: Perform Chinese field matching;

[0019] First, a knowledge base is used to perform preliminary matching on obviously equivalent special fields; then, Chinese automatic word segmentation technology is used to segment the character sequences after preliminary matching to form character sequences containing word segmentation tags.

[0020] Steps 2-3 involve calculating similarity based on edit distance;

[0021] Edit distance (m+1)×(n+1) order matrix D ij The calculation formula is as follows:

[0022]

[0023] In the formula, D ij =D(s1…s i ,t1…t i ), 0≤i≤m, 0≤j≤n, D ijdenotes the edit distance from {s1…s i} to {t1…t i}, s refers to the source string, and t refers to the target string;

[0024] Step 2-4: Data cleaning based on improved N-Gram algorithm;

[0025] Field matching using a sliding window of dynamic size, setting the initial window size N0, and the size of the subsequent window is calculated during execution;

[0026] Window size calculation method: Let the current window being processed be N1, r1 and r l represent the first and last records in the window, respectively, and the distance d t between them satisfies:

[0027]

[0028] At this time, the average distance between records in the window N1 is d t (r1,r l ) / N1;

[0029] The size of the next window N2 is: is the minimum threshold value of the window.

[0030] Based on the above, the method for identifying important targets in step 3 uses overlapping community detection technology:

[0031] Define the graph G = (V, E) to be processed, which consists of a finite number of nodes V and edges E between nodes, and satisfies: for any (u, v) ∈ E, u ≠ v; (u, v) ∈ E, if and only if (v, u) ∈ E, and the final detected cluster is S', then the specific steps of generating S' from G are as follows:

[0032] Step 3-1: For each node u in the graph G, use the local clustering algorithm to divide the ego-net of node u:

[0033]

[0034] where t u = n p (A l , G[N u ]) and n p represents the number of sets;

[0035] Step 3-2: Create a copy set V';

[0036] Each node u in V corresponds to t u copies in V', denoted as u​i , i = 1, 2,..., t u ;

[0037] Step 3-3: add edges between copies;

[0038] If (u, v) E, and Then add an edge (u i , v j ) in E';

[0039] Step 3-4: cluster on G' = (V', E') using the global clustering algorithm Ag to obtain the partition S" of V';

[0040] Step 3-5: for each set C' E S", associate a cluster consisting of the corresponding nodes of V That is Output S' = {C(C') | C' E S"}.

[0041] Based on the above, the user blog vector representation in step 4 is as follows:

[0042] After the blog is segmented, the blog presents a word table of size n, and the i-th semantic word in the table is represented as:

[0043] word i = (site i , vector i )

[0044] In the formula, site i represents the position of the semantic word in the semantic vector dictionary, and vector i represents the semantic vector corresponding to the semantic word;

[0045] The initial vector representation of the blog composed of n semantic words is:

[0046] Blog = (word1, word2, word3,..., word n )

[0047] Select a blog Blog1 of a source user and a blog Blog2 of a candidate user to be compared, and use the average pooling method to obtain a fixed dimension blog vector V1 of the source user, the formula is as follows:

[0048]

[0049]

[0050] In the formula, n1 represents the number of semantic words in the source user blog, and a represents the weight of each semantic word.

[0051] The weighted average pooling calculation method is used to obtain a candidate user blog vector V2 with the same dimension as the source user blog vector:

[0052]

[0053]

[0054]

[0055] In the formula, β represents the fluctuation amplitude of the weight of the same semantic word, and γ i represents the weight of the i-th semantic word.

[0056] Based on the above, the user personal information vectorization content in step 5 is as follows:

[0057] The public feature information with five dimensions of recognition degree is represented and one-hot encoded: gender, location province, location city, constellation, and login method; Since the possible values of province and city are more, the province is numbered first to form a dictionary, and then the user province is replaced with the corresponding number in the dictionary to realize digitization, and the city is digitized in the same way. Then, according to the sample data, use OneHotEncoder in python's sklearn to one-hot encode the user personal information, and obtain the source user personal information vector I1={i 11 ,i 12 ,i 13 ,i 14 ,i 15} and the candidate user personal information vector I2={i 21 ,i 22 ,i 23 ,i 24 ,i 25}.

[0058] Based on the above, the blog similarity calculation formula of the source user and the candidate user in step 6 is as follows:

[0059]

[0060] The cosine similarity calculation is used to calculate the personal information similarity of the source user and the candidate user, and the calculation formula is as follows:

[0061]

[0062] After the blog similarity and the personal information similarity of the source user and the candidate user are calculated respectively, next, the weight δ of the user personal information is found, so that the weighted result is most accurate, that is, the character portrait can be described; wherein, the weighted similarity is expressed as:

[0063] S(P1,P2)=δ×S(I1,I2)+(1-δ)×S(V1,V2).

[0064] The present application has outstanding substantial features and significant progress compared with the prior art, specifically:

[0065] (1) The present application can well process the case data;

[0066] (2) In the present application, the important target recognition technology based on overlapping community discovery is adopted, the local clustering structure is used for guiding to detect the overlapping community, and the complex overlapping clustering problem is reduced to a simpler and more easily handled non-overlapping partition problem;

[0067] (3) The present application uses vector representation, behavior association and other methods to construct a network user identity association model based on multi-dimensional features, and mines the cross-social platform user association relationship and restores the user social relationship network. DETAILED DESCRIPTION

[0068] The technical solutions of the present application are further described in detail through specific embodiments.

[0069] Embodiment 1

[0070] The present embodiment provides a case big data network identity association and character portrait method, the method comprises:

[0071] Step 1, obtaining case data; wherein, the case data comprises: user ID, user nickname, IP address, superior and real name.

[0072] Step 2, cleaning the obtained data;

[0073] The specific cleaning method is:

[0074] Step 2-1, if there is blank content in the data, the data is deleted; if there are multiple attribute items completely repeated in the data, only one record is kept and the rest is deleted;

[0075] Step 2-2, Chinese field matching;

[0076] First, the knowledge base is used to preliminarily match the special fields that are obviously equivalent; then, the Chinese automatic word segmentation technology is used to perform word segmentation processing on the character sequence after the preliminary matching, so as to form a character sequence containing word segmentation marks; wherein the knowledge base refers to an existing open source Chinese vocabulary library;

[0077] Step 2-3, similarity calculation based on edit distance is performed;

[0078] The calculation formula of the edit distance (m+1) x (n+1) order matrix D is as follows: ij

[0079]

[0080] In the formula, D ij = D(s1…s i , t1…t i ), 0≤i≤m, 0≤j≤n, D ij represents the edit distance from {s1…s i} to {t1…t i}, s refers to the source string, and t refers to the target string;

[0081] Step 2-4, data cleaning based on the improved N-Gram algorithm;

[0082] In the improved N-Gram algorithm, a sliding window of a dynamic size is used for field matching; the window size is determined by the current window size N i , the distance d t between the first and last records in the window, and the window minimum threshold ; the initial value N0 of the window size is set when the algorithm is running, and the size of the subsequent window is obtained during the algorithm execution process;

[0083] The window size calculation method is as follows: N1 is recorded as the window currently being processed, r1 and r l represent the first and last records in the window respectively, and the distance d t between the two satisfies:

[0084]

[0085] At this time, the average distance between the records in the window N1 is d t (r1,r l ) / N1;

[0086] The size N2 of the next window is: is the window minimum threshold.

[0087] Step 3, important targets are identified by using the overlapping community discovery technology;

[0088] ​A specific method of identifying important targets:

[0089] The discovery of overlapping communities mainly includes two steps: local ego-net analysis and global graph partitioning. In the first step, an ego-net is created for each node u, and then the neighbor nodes of node u are separated out. For each community, a new node u is created, and each copy is uniquely associated with a cluster in the part of nodes. Then, each edge between nodes in the original graph is mapped to the copies. In the second step of global graph partitioning, a graph partitioning algorithm is used to partition the copy graph and output the detected clusters;

[0090] The graph G=(V,E) to be processed is defined by a finite number of nodes V and edges between nodes , and satisfies: for any (u,v)∈E, u≠v; (u,v)∈E, if and only if (v,u)∈E, and the final detected clusters are S', then the specific steps of generating S' from G are as follows:

[0091] Step 3-1: For each node u in the graph G, use a local clustering algorithm to partition the ego-net of node u:

[0092]

[0093] Where t u =n p (A l ,G[N u ]) and n p represents the number of sets;

[0094] Step 3-2: Create a copy set V';

[0095] Each node u in V corresponds to t u copies in V', denoted as u i , i=1,2,…, t u ;

[0096] Step 3-3: Add edges between copies;

[0097] If (u,v)∈E, and Then add an edge (u i ,v j ) to E';

[0098] Step 3-4: Use the global clustering algorithm Ag to cluster on G'=(V',E'), and obtain the partition S'' of V';

[0099] Step 3-5: For each set C' in S", associate a cluster consisting of the corresponding nodes of V That is Output S' = {C(C') | C' ∈ S"}.

[0100] Step 4, user blog vectorization;

[0101] Due to the randomness of blog expression and the complexity of data sources, it is necessary to quantitatively analyze the blog. The vectorization method is adopted. First, the blog data is segmented, then the semantic words are replaced by the corresponding semantic vectors in the dictionary, and then the blog vector representation model based on convolutional neural network is used to realize the vectorization of user blog, so as to facilitate the similarity comparison between source users and candidate users;

[0102] The specific user blog vectorization representation is as follows:

[0103] After the blog is segmented, the blog presents a word table with a size of n, and the i-th semantic word in the table is represented as:

[0104] word i =(site i ,vector i )

[0105] In the formula, site i represents the position of the semantic word in the semantic vector dictionary, and vector i represents the semantic vector corresponding to the semantic word;

[0106] The initial vector representation of the blog composed of n semantic words is:

[0107] Blog=(word1,word2,word3,…,word n )

[0108] Select a blog Blog1 of the source user and a blog Blog2 of the candidate user to be compared, and build a model to compare their semantics;

[0109] Since the source user blog is the compared object, each semantic word in the source user blog is an important measure of the user blog to be compared, so all the semantic words in the blog are used as core words, and the average pooling method is used to obtain the fixed dimension blog vector V1 of the source user, and the formula is as follows:

[0110]

[0111]

[0112] In the formula, n1 represents the number of semantic words in the source user blog, and a represents the weight of each semantic word.

[0113] As the candidate user blog to be compared, the same or similar semantic words in the blog as the focus, so as to increase the weight of this part of the semantic words, using the weighted average pooling calculation method, obtain the same dimension of the source user blog vector candidate user blog vector V2:

[0114]

[0115]

[0116]

[0117] In the formula, β represents the fluctuation amplitude of the weight of the same semantic word, and γ i represents the weight of the i-th semantic word.

[0118] Step 5, user personal information vectorization;

[0119] The multi-dimensionality and discontinuity of personal information cannot be directly used as the similarity calculation of user identity association, and the user personal information needs to be encoded and vectorized. The user personal information is one-hot encoded, and the user personal information is converted into a sparse vector representation.

[0120] The specific user personal information vectorization content is as follows:

[0121] The five-dimensional public feature information with recognition degree is represented and one-hot encoded: gender, location province, location city, constellation and login method. Since the possible values of province and city are more, the province is numbered to form a dictionary first, and then the user province is replaced with the corresponding number in the dictionary to realize digitization. Similarly, the city is digitized. Then, according to the sample data, the user personal information is one-hot encoded using OneHotEncoder in python, and the source user personal information vector I1={i 11 , i 12 , i 13 , i 14 , i 15} and the candidate user personal information vector I2={i 21 , i 22 , i 23 , i 24 , i 25} are obtained.

[0122] Step 6, source user and candidate user blog similarity calculation;

[0123] After the personal information and the writing behavior of the user are represented by vectors, the similarity between the source user and the candidate user can be calculated, and then the comprehensive similarity between the users can be calculated.

[0124] The blog similarity calculation formula of the source user and the candidate user is as follows:

[0125]

[0126] The personal information similarity of the source user and the candidate user is calculated by using the cosine similarity calculation, and the calculation formula is as follows:

[0127]

[0128] After the blog similarity and the personal information similarity of the source user and the candidate user are calculated respectively, the appropriate weight δ of the user personal information is found, so that the weighted result is most accurate, and the character portrait can be described; wherein the weighted similarity is expressed as:

[0129] S (P1, P2) = δ × S (I1, I2) + (1- δ) × S (V1, V2).

[0130] Finally, it should be noted that: the above examples are used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: the specific embodiments of the present application can be modified or some technical features can be replaced by equivalent; without departing from the spirit of the technical solutions of the present application, they should be covered in the technical solution range of the present application claimed.

Claims

1. A method for linking network identities and creating character profiles based on big data in case studies, characterized in that: The method includes: Step 1: Obtain case data; Step 2: Clean the acquired data; Data cleaning includes: Step 2-1: If there are blank entries in the data, delete that entry; if there are multiple records with completely duplicate attribute items, keep only one record and delete the rest. Step 2-2: Perform Chinese field matching; First, a knowledge base is used to perform preliminary matching on obviously equivalent special fields; then, Chinese automatic word segmentation technology is used to segment the character sequences after preliminary matching to form character sequences containing word segmentation tags. Steps 2-3 involve calculating similarity based on edit distance; Edit distance (m+1)×(n+1) order matrix D ij The calculation formula is as follows: ， In the formula, D ij =D(s1…s i ,t1…t i ), 0≤i≤m, 0≤j≤n, D ij Indicates from {s1…s i }, to {t1…t i The edit distance of}, where s refers to the source string and t refers to the target string; Steps 2-4: Data cleaning based on the improved N-Gram algorithm; Field matching is performed using a dynamically sized sliding window. The initial window size is set to N0, and the subsequent window size is calculated as the execution process progresses. Window size calculation method: Let the window currently being processed be N1, r1 and r l Let represent the first and last records in the window, respectively. Then the distance d between them is... t satisfy: d t (r1,r l )<φ At this time, the average distance between all records in window N1 is d. t (r1,r l ) / N1; The next step is to set the window size N2 as follows: *N2=φ, where φ is the minimum threshold of the window. Step 3: Identify important targets using overlapping community detection technology; A method for identifying important targets using overlapping community detection techniques: Define the graph G=(V,E) to be processed as consisting of a finite number of nodes V and the edges between the nodes. The clusters are composed of G and satisfy the following conditions: for any (u,v)∈E, u≠v; (u,v)∈E, if and only if (v,u)∈E, the final detected cluster is S'. The specific steps for generating S' from G are as follows: Step 3-1: For each node u in graph G, use a local clustering algorithm to partition the ego-net of node u: ， Among them, t u =n p (A l ,G[N u ]), n p Indicates the number of sets; Step 3-2: Create a replica set V'; Each node u in V corresponds to t in V' u Each copy is represented as u i i=1,2,…,t u ; Step 3-3: Add edges between replicas; If (u,v)∈E, and Then add an edge (u) to E' i ,v j ); Steps 3-4: Perform clustering on G'=(V',E') using the global clustering algorithm Ag to obtain the partition S'' of V'; Steps 3-5: For each set C'∈S'' in the partition of V', associate it with a cluster consisting of the corresponding nodes of V. That is, C(C')={u∈V| i stu i ∈C'}; Output S'={C(C')|C'∈S''}; Step 4: Vectorize user blog posts; The vectorized representation of a user's blog post is as follows: After word segmentation, the blog post presents a word list of size n, where the i-th semantic word is represented as: Word i =(site i ,vector i ) In the formula, site i The vector represents the position of a semantic word in the semantic vector dictionary. i This represents the semantic vector corresponding to a semantic word; The initial vector representation of a blog post, consisting of tuples of n semantic words, is as follows: Blog=(Word1,Word2,Word3,…,Word n ,) Select a blog post (Blog1) from the source user and a blog post (Blog2) from the candidate user to be compared. Use average pooling to obtain a fixed-dimensional blog post vector (V1) for the source user, as shown in the following formula: ， ， In the formula, n1 represents the number of semantic words in the source user's blog post, and α represents the weight of each semantic word; Using a weighted average pooling method, candidate user blog post vectors V2 with the same dimensions as the source user blog post vectors are obtained. ， In the formula, β represents the fluctuation range of the weight of the same semantic word, and γ i This represents the weight of the i-th semantic word; Step 5: Vectorize user personal information; The vectorized content of user personal information is as follows: Five distinctive dimensions of publicly available feature information are represented and one-hot encoded: gender, province, city, zodiac sign, and login method. Since there are many possible values ​​for province and city, provinces are first numbered to form a dictionary. Then, the user's province is replaced with the corresponding number from the dictionary to achieve digitization. Similarly, cities are digitized. Based on sample data, the OneHotEncoder from sklearn in Python is used to perform one-hot encoding on the user's personal information, obtaining the source user personal information vector I1={i 11 i 12 i 13 i 14 i 15 } and candidate user personal information vector I2={i 21 i 22 i 23 i 24 i 25 }; Step 6: Calculate the similarity between the blog posts of the source user and the candidate user; The formula for calculating the similarity between blog posts from source users and candidate users is as follows: ， The cosine similarity is used to calculate the similarity of personal information between the source user and the candidate user. The calculation formula is as follows: ， After calculating the blog post similarity and personal information similarity between the source user and the candidate user, the next step is to find an appropriate weight δ for the user's personal information to maximize the accuracy of the weighted result, thus creating a user profile. The weighted similarity is expressed as: S(P1,P2)=δ×S(I1,I2)+(1-δ)×S(V1,V2).

2. The method for case big data network identity association and character profiling according to claim 1, characterized in that, The case data in Step 1 includes: User ID, User nickname, IP address, Supervisor, and Real Name.

Images