Cross-domain clue discovery method based on multi-source data clustering analysis

Abstract

The invention discloses a cross-domain clue discovery method based on multi-source data clustering analysis. The method comprises the following steps: acquiring sample data; performing clustering analysis on the sample data based on a probability model to obtain a cross-domain feature classification target and recommendation; the probability model refers to a classification scheme with the maximum posterior probability; the classification scheme refers to a model for calculating an N probability according to the sample data; the N probability is the probability of generating a feature word sequence with the length of N; the posterior probability is a probability obtained by calculating the classification scheme based on a preset posterior model. According to the method, target attribute analysis and judgment are realized, a feature clustering model is constructed to realize target analysis and identification of specific behavior features, and feature classification target discovery and recommendation are completed based on an identification result.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a cross-domain clue discovery method based on multi-source data clustering analysis. Background Art

[0002] The gradual networking and virtualization of real-world data has led to a rapid increase in the amount of data on the Internet. With the continuous accumulation of data, the data on the Internet contains a large amount of inherent knowledge, which can help people complete a series of intelligent applications such as data analysis and decision support. People are increasingly eager to obtain information and knowledge from these large amounts of data to reveal implicit, previously unknown and potentially valuable information. Data mining-related fields have once again become one of the current hot research directions. Cluster analysis is a very active research field in data mining. Simply put, it is a process of analyzing a set of physical or abstract objects, classifying objects with similar features into the same class, and objects with dissimilar features into different classes. Clustering algorithms play an important role in pattern recognition, intelligent search, bioinformatics, and semantic disambiguation. Existing clustering analysis models cannot well analyze the information hidden in text data that is not easy to directly obtain statistically, especially to better discover clues and form efficient and usable cross-domain traffic metadata. Summary of the invention

[0003] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a cross-domain clue discovery method based on multi-source data cluster analysis, realize target attribute analysis and verification, and build a feature clustering model to realize target analysis and identification of specific behavioral characteristics, and complete feature classification target discovery and recommendation based on the identification results. In order to achieve the above-mentioned purpose and other advantages according to the present invention, a cross-domain clue discovery method based on multi-source data cluster analysis is provided, comprising:

[0004] Get sample data;

[0005] Performing cluster analysis on the sample data based on a probability model to obtain cross-domain feature classification targets and recommendations;

[0006] The probability model refers to a classification scheme of maximum a posteriori probability;

[0007] The classification scheme refers to a model for calculating N probabilities based on the sample data;

[0008] The N probability is the probability of generating a feature word sequence of length N;

[0009] The posterior probability refers to the probability calculated by the classification scheme based on a preset posterior model.

[0010] Preferably, the posterior probability calculation method comprises:

[0011] Calculate each potential partitioning scheme Ω * The posterior probability of the potential partitioning scheme Ω * The posterior probability is:

[0012] P(Ω * |D)∝∏ d∈D P(ψ (d) )P(d|c(d)) (1),

[0013] When calculating, formula (1) can be transformed into:

[0014]

[0015] Among them, d is the article, K is the expected number of categories, k is a category, and D is the sample data.

[0016] Preferably, the probability of generating a feature word sequence of length N in the classification scheme is:

[0017]

[0018] Among them, π(N|c k ) means c k The probability of generating a sequence of length N; the length of the article is independent of the category, so the length of the article is a constant, and then the probability of generating a feature word sequence of length N in the classification scheme is finally:

[0019]

[0020] Preferably, the potential partitioning scheme Ω * The posterior probability of simplifies to:

[0021]

[0022] Among them, |c k | indicates category c k The number of documents in ; which implicitly contains constraints and and

[0023] A cross-domain clue discovery device based on multi-source data cluster analysis, comprising:

[0024] A receiving module, used for acquiring sample data;

[0025] Model selection module, used for the selection of probability models;

[0026] The calculation module is used to calculate different classification schemes through the posterior model to obtain multiple posterior probabilities and the specific form of the classification scheme is the probability of generating a feature word sequence with a length of N in a classification scheme.

[0027] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements a cross-domain clue discovery method based on multi-source data clustering analysis when executing the computer program.

[0028] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a cross-domain clue discovery method based on multi-source data clustering analysis.

[0029] Compared with the prior art, the present invention has the following beneficial effects: by clustering analysis, all samples or variables are respectively aggregated into different classes, so that individuals in the same class have greater similarity and individuals in different classes have greater differences, target attribute analysis and verification are achieved, and a feature clustering model is constructed to achieve target analysis and identification of specific behavioral features, and feature classification target discovery and recommendation are completed based on the identification results. The complexity of parameter learning is greatly reduced through the posterior model, and the accurate maximum a posteriori estimation can be directly solved, avoiding the complexity and amount of calculation caused by the likelihood model using the EM algorithm to estimate the maximum likelihood probability. Furthermore, at this point, the error caused by the EM algorithm falling into the local maximum is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a process of a cross-domain clue discovery method based on multi-source data clustering analysis according to the present invention;

[0031] Figure 2 A schematic diagram of a flow chart in an embodiment of a cross-domain clue discovery method based on multi-source data clustering analysis according to the present invention;

[0032] Figure 3 It is a schematic diagram of the results of splitting and agglomerating clusters formed by the hierarchical clustering method of the cross-domain clue discovery method based on multi-source data cluster analysis according to the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Reference Figure 1-3, a cross-domain clue discovery method based on multi-source data clustering analysis, comprising:

[0035] Get sample data;

[0036] Performing cluster analysis on the sample data based on a probability model to obtain cross-domain feature classification targets and recommendations;

[0037] The probability model refers to a classification scheme of maximum a posteriori probability;

[0038] The classification scheme refers to a model for calculating N probabilities based on the sample data;

[0039] The N probability is the probability of generating a feature word sequence of length N;

[0040] The posterior probability refers to the probability calculated by the classification scheme based on a preset posterior model.

[0041] Specifically, in this embodiment, a probability model is selected for sample data to perform cluster analysis, and one selection method is as follows:

[0042] In cluster analysis based on probability models, it is generally assumed that the data is generated through a mixture model, and each component in the model is regarded as a category. Mixture models assume that the distribution density function of the data is a linear combination of some simple distribution density functions, that is:

[0043]

[0044] p i is the weight of each distribution, f i (x) is the density function of the component distribution. The choice of component distribution largely determines the adaptability of the mixture model to different data. Choosing the right distribution family has a great impact on the effect of the mixture model.

[0045] In cluster analysis using a mixture model, if the number of classification categories is known to be K, the mixture model can be expressed as a mixture of K component distributions:

[0046]

[0047] Among them, M is the mixed model, M k represents the component corresponding to the kth category. P(d|M k ) represents the probability that category k generates data d, P(M k ) represents the component P(d|M k ) can be understood as the prior probability that a data from the entire data set (rather than a given sample) is exactly of category k.

[0048] If the given data D={d1,d2,…,d N} is a random sample, where each instance d i ,d j are all independent of each other. The likelihood function of the above mixed model M for sample D can be expressed as:

[0049]

[0050] The model selection problem is usually transformed into L(D|M) often contains some parameters related to M that need to be determined, and the maximum likelihood estimation can generally be used, that is, It is usually very difficult to solve the maximum likelihood estimate of the mixed model through analytical methods, and some numerical methods, such as the expectation maximization algorithm (EM), are needed to obtain these parameters.

[0051] The basic EM algorithm is an iterative process that alternately executes the E step and the M step to increase the value of the likelihood function. Its general process can be described as:

[0052] Step E: Based on the current parameter estimates, calculate the conditional expectation of the likelihood function for the potential data after adding the potential data;

[0053] In the M step, the current parameter estimate is maximized based on the expected value of the E step;

[0054] For the corrected parameter estimates, calculate the likelihood function value. If the likelihood function value reaches the pre-set threshold or the specified number of iterations, stop, otherwise go to 1).

[0055] To facilitate the explanation of the EM algorithm for the mixture model, we rewrite the form of the mixture model and omit M to simplify the writing: Where Θ is the parameter set of the model M, and the parameter σ k =P(M k ), θ k is P(d|M k ) and the model component M k The associated parameters.

[0056] To use the EM algorithm, add a latent vector indicating the category to each data d and: Obviously (d) The value range is {Ψ1,Ψ2,…,Ψ K The log-likelihood of model M given the added data of data sample set D (a matrix of order |D|×K, denoted as ψ) is expressed as:

[0057]

[0058] Where c(d) represents ψ (d) The category assigned to document d, that is, ψ (d) The only component in is 1. Given the current parameters The distribution of ψ can be expressed as in:

[0059]

[0060] Compute logL(D|ψ,Θ) for a given sample data D and given parameters Conditional expectation on:

[0061]

[0062] The EM algorithm for the mixed model can be implemented in the following two steps:

[0063] Step E, at the current model parameters Calculate

[0064] M steps, through Update the parameters in the model.

[0065] It can be proved that in the new Under this condition, the likelihood function value of the model for the given data will be improved. Generally, after giving a specific distribution form, Each element σ k ,θ k The estimated value of can be obtained analytically by the Lagrange multiplier method.

[0066] Specifically, in this embodiment, a probability model is selected for sample data to perform cluster analysis, and one selection method is as follows:

[0067] Different classification schemes are used through the posterior model to obtain multiple posterior probabilities, and the multiple posterior probabilities are compared to obtain the classification scheme with the maximum posterior probability.

[0068] To represent the classification scheme Ω, we consider that each article d has a corresponding probability vector ψ (d) , K is the expected number of categories. Thus, the data D corresponds to a matrix ψ with |D| rows and K columns, and each of its elements Indicates the extent to which article d belongs to category c k , so ψ can represent any classification scheme. The posterior probability of Ω can be expressed as:

[0069]

[0070] Assuming that the generation process of each article is independent of each other, and the corresponding category vector of each article is also independent of each other, P(ψ|D) can be expressed as:

[0071]

[0072] Usually, the total probability formula can be given This is exactly the mixed model of the document, and the above formula is difficult to handle mathematically. Since non-fuzzy clustering is used in this application, each potential partitioning scheme considered by the clustering algorithm is * Each document is classified into a specific category, that is, Ω * The class vector ψ (d) Only one component can be 1, and all others can be zero: In this case ψ (d) The only possible values ​​are:

[0073]

[0074] For satisfying ψ (d) The category vector of ∈Ψ obviously has P(d|ψ (d) )=P(d|c(d)), where c(d) represents the category vector ψ (d) The only non-zero component in . In this way, we can give every potential partitioning scheme Ω * The posterior probability is:

[0075]

[0076] and In comparison, P(Ω * |D) is very easy to calculate. By changing the form, we can get:

[0077]

[0078] Compared with the usual likelihood model, the use of this posterior model greatly reduces the complexity of parameter learning, and can directly solve the accurate maximum a posteriori estimate, avoiding the complexity and amount of calculation caused by the likelihood model using the EM algorithm to estimate the maximum likelihood probability. Furthermore, at this point, the error caused by the EM algorithm falling into the local maximum is avoided.

[0079] Specifically, in this embodiment, the posterior model calculates the posterior probability through a specific form of the classification scheme, and the specific form of the classification scheme is the probability of generating a feature word sequence of length N in a classification scheme, which is as follows:

[0080] In order to obtain P(d|ck ), we describe the process of generating feature words in a document to give a reasonable random occurrence model of document feature sequences. According to the vector space model (VSM), each article can be represented by a vector in a feature space. Each dimension of the vector represents a feature word, and its value represents the number of times the feature word appears in the article. This representation method ignores the structural information of the article, making clustering completely dependent on the frequency of occurrence of features in the document. In VSM:

[0081] The feature vector of document d is represented by d=(n1,n2,…,n m ,…n M-1 ,n M ), 1≤m≤M, where n m Represents the feature word t of the mth dimension in the feature space m Frequency of occurrence.

[0082] Document class c is a collection of documents in the document set D, and its feature vector can be expressed as in Represents the feature word t in document d m Frequency of occurrence.

[0083] Taking into account the characteristics of the VSM model (vector space model), each document is represented only by the frequency of each feature, and is independent of the order in which the features appear. The document class can be intuitively regarded as a feature generator, and each feature has a different probability of occurrence in different document classes.

[0084] The document class c k As a feature (symbol) generator, by observing c k The document is obtained by generating a sequence of features. Since VSM ignores the order in which the features appear, we can consider the frequency of each feature to be an independent and identically distributed (iid) random variable. k Generate t m The probability can be expressed as c k Medium m The frequency of occurrence, that is:

[0085]

[0086] Define the parameters θ of the document class (k) is the same as the distribution P(d|c k ) is a continuous probability vector associated with: and Actually It means document class c k Generate feature word t m The probability P(tm |c k ).

[0087] The generation of each feature word in a document class is an independent process. i Indicates the i-th feature word appearing in document d, document class c k The probability of generating a feature word sequence d of length N can be expressed as:

[0088]

[0089] Where π(N|c k ) means c k The probability of generating a sequence of length N. We believe that the length of the article has nothing to do with the category, so we regard it as a constant. This gives P(d|c k )'s final form:

[0090]

[0091] Based on this model, the potential partitioning scheme Ω * The posterior probability can be expressed as:

[0092]

[0093] Where |c k | indicates category c k The number of documents in . This implicitly contains the constraint and and

[0094] Specifically, in this embodiment, a cross-domain clue discovery method based on multi-source data clustering analysis uses a hierarchical clustering algorithm, specifically including:

[0095] Each time, a pair of categories generated in the previous step is selected to be merged into one category. The selection is based on the partitioning scheme generated in this step having the largest posterior probability P(Ω * |D). For the convenience of writing, we omit Ω * The asterisk on . The local objective function using maximum a posteriori estimation is:

[0096]

[0097] Each step in hierarchical clustering is a local optimization based on the selection of the previous step. Considering that the kth step has been completed, the classification scheme generated by the algorithm is Ω k ={c1,c2,…,c |D|-k}, the corresponding posterior probability P(Ω k |D) is a constant that has been determined, and obviously

[0098]

[0099] After rearranging the right formula, we get:

[0100]

[0101] In the above formula, c z =c x ∪c y , f m Represents the characteristic word t m The frequency of occurrence in the entire category. x ,c y Merge them into one category, and the k+1 steps of aggregation are completed.

[0102] The number of devices and processing scales described here are used to simplify the description of the present invention, and the application, modification and variation of the present invention will be obvious to those skilled in the art.

[0103] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A cross-domain clue discovery method based on multi-source data clustering analysis, characterized in that: include: Get sample data; Performing cluster analysis on the sample data based on a probability model to obtain cross-domain feature classification targets and recommendations; The probability model refers to a classification scheme of maximum a posteriori probability; The classification scheme refers to a model for calculating N probabilities based on the sample data; The N probability is the probability of generating a feature word sequence of length N; The posterior probability refers to the probability calculated by the classification scheme based on a preset posterior model.

2. A cross-domain clue discovery method based on multi-source data clustering analysis as claimed in claim 1, characterized in that: The posterior probability calculation method comprises: Calculate each potential partitioning scheme Ω * The posterior probability of the potential partitioning scheme Ω * The posterior probability is: P(Ω * |D)∝∏ d∈D P(ψ (d) )P(d|c(d)) (1), When calculating, formula (1) can be transformed into: Among them, d is the article, K is the expected number of categories, k is a category, and D is the sample data.

3. The cross-domain clue discovery method based on multi-source data clustering analysis according to claim 1, characterized in that: The probability of generating a feature word sequence of length N in the classification scheme is: Among them, π(N|c k ) means c k The probability of generating a sequence of length N; the length of the article is independent of the category, so the length of the article is a constant, and then the probability of generating a feature word sequence of length N in the classification scheme is finally:

4. The cross-domain clue discovery method based on multi-source data clustering analysis according to claim 2, characterized in that: The potential partitioning scheme Ω * The posterior probability of simplifies to: Among them, |c k | indicates category c k The number of documents in ; which implicitly contains constraints and and 5. A cross-domain clue discovery device based on multi-source data cluster analysis, characterized in that: include: A receiving module, used for acquiring sample data; Model selection module, used for the selection of probability models; The calculation module is used to calculate different classification schemes through the posterior model to obtain multiple posterior probabilities and the specific form of the classification scheme is the probability of generating a feature word sequence with a length of N in a classification scheme.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images