A method and system for intelligent analysis based on multi-source data

Abstract

The application relates to the technical field of encrypted communication and discloses a method and system for intelligent analysis based on multi-source data, wherein the method for intelligent analysis based on multi-source data comprises the following steps: obtaining multi-dimensional metadata of an encrypted communication flow to generate a characteristic vector set; using a Markov chain model to calculate state transition probability of a communication session, dividing the communication session according to a time window and mapping the communication session to discrete states to generate a session state evolution sequence; analyzing the session state evolution sequence based on a hierarchical topic model, mapping the state sequence to a behavior topic through a latent Dirichlet allocation, mapping the behavior topic to an intention category through a hierarchical Dirichlet process, and outputting a communication intention category and a confidence score; the application overcomes the limitation that static feature analysis cannot reflect the complete life cycle of a session, solves the semantic gap problem caused by limited metadata information dimensions, and improves the accuracy and robustness of intention recognition.

Description

Technical Field

[0001] This invention relates to the field of encrypted communication technology, and more specifically, to a method and system based on intelligent analysis of multi-source data. Background Technology

[0002] In enterprise network environments, various applications and services widely use encrypted communication protocols, including normal business communications (such as encrypted file transfers, video conferencing, and cloud service access), but also potentially concealing malicious activities (such as data breaches, remote control, and covert channel communication). Security teams need to accurately identify the types of communication intent by analyzing the evolution of communication behavior without decryption, thereby enabling precise enforcement of security policies.

[0003] Encrypted communication content is invisible; only limited metadata information (packet size, time interval, certificate, etc.) can be obtained. Communication patterns of different applications may appear similar but have very different intentions. At the same time, communication intentions evolve dynamically as the session progresses (e.g., from authentication to data transmission to heartbeat maintenance), and single-point analysis cannot capture the complete intention trajectory. Furthermore, advanced attackers will deliberately simulate the communication patterns of normal applications, rendering detection methods based on static features ineffective. Summary of the Invention

[0004] This invention provides a method and system based on intelligent analysis of multi-source data, which solves the technical problem of semantic gap caused by the limited dimension of metadata information in related technologies.

[0005] This invention provides a method for intelligent analysis based on multi-source data, comprising the following steps:

[0006] Obtain multidimensional metadata of the encrypted communication stream and generate a set of feature vectors;

[0007] The state transition probability of a communication session is calculated using a Markov chain model. The communication session is divided into time windows and mapped to discrete states to generate a session state evolution sequence.

[0008] Based on the hierarchical topic model, the session state evolution sequence is analyzed. The state sequence is mapped to behavioral topics through latent Dirichlet assignment, and the behavioral topics are mapped to intent categories through hierarchical Dirichlet process. The communication intent category and confidence score are output.

[0009] The state transition probability is calculated as follows: the probability of transitioning from state i to state j is equal to the number of observations of transitioning from state i to state j divided by the sum of the number of observations of transitioning from state i to all states.

[0010] Furthermore, the step of obtaining multidimensional metadata of the encrypted communication stream and generating a feature vector set specifically includes:

[0011] Obtain multidimensional metadata sequences of encrypted communication streams from network traffic acquisition devices, including packet length sequences, packet arrival time interval sequences, TLS handshake information, certificate chain data, and DNS query records;

[0012] The acquired metadata is preprocessed, including: one-hot encoding of fields such as handshake type and cipher suite in TLS handshake information; label encoding of fields such as certificate issuer and validity period in certificate chain data; mapping domain names in DNS query records to numerical identifiers using hash functions; and scaling packet length and arrival time interval to the range of 0 to 1 using the minimum-maximum normalization method.

[0013] Feature extraction is performed to generate temporal feature vectors and statistical feature vectors. The temporal feature vectors include the time series distribution of packet length and the change pattern of arrival time intervals, while the statistical feature vectors include statistical indicators such as average packet length, packet length variance, flow duration, and total number of packets.

[0014] Furthermore, the step of using a Markov chain model to calculate the state transition probability of a communication session and generate a session state evolution sequence specifically includes:

[0015] The communication session is divided into fixed time windows, and the feature vectors within each time window are aggregated and statistically analyzed.

[0016] The aggregated feature vectors are standardized using the Z-score standardization method. The standardized value is equal to the original value minus the mean and then divided by the standard deviation.

[0017] The standardized feature vectors are clustered using K-means to map them to n discrete states, and the optimal number of states is selected by the silhouette coefficient.

[0018] Count the frequency of state transitions between adjacent time windows and construct a state transition frequency matrix;

[0019] The transition frequency matrix is ​​normalized to obtain the state transition probability matrix. Based on this matrix and the initial state distribution, a session state evolution sequence is generated.

[0020] Furthermore, it also includes:

[0021] A variational autoencoder is used to map discrete states to a continuous semantic space, generating a continuous representation of the intent trajectory;

[0022] The variational autoencoder includes an encoder and a decoder. The encoder maps the session state evolution sequence to the distribution of latent variables, and the decoder reconstructs the session state evolution sequence from the latent variables.

[0023] The semantic representation vectors for different intent categories are learned by optimizing the variational lower bound, which includes a reconstruction loss and a KL divergence regularization term. The reconstruction loss is calculated as cross-entropy, and the KL divergence regularization term constrains the difference between the latent variable distribution and the prior distribution.

[0024] Furthermore, it also includes:

[0025] Graph neural networks are used to analyze the communication context and generate relationship embedding vectors.

[0026] Construct a communication relationship graph between source and destination IP pairs, where nodes represent IP addresses and edges represent historical communication relationships, with communication frequency, duration, and data volume as attributes of the edges;

[0027] The graph attention network is used to learn the representations of nodes and edges, and the relation embedding vector of each IP pair is generated by aggregating the neighbor node information.

[0028] The graph attention network calculates the attention coefficient between each node and its neighboring nodes, and obtains the output features of the node by weighted aggregation of the neighboring node features based on the attention coefficient.

[0029] Furthermore, it also includes:

[0030] Based on multi-view learning and fusion of multi-dimensional features, the final communication intent category is output;

[0031] The continuous representation of intent trajectory, session feature vector, and relation embedding vector are used as three views. Through collaborative training, the representations of different views are aligned in a shared semantic space.

[0032] The three views are mapped to view representations using a view-specific encoder and projected onto a shared semantic space through a shared mapping layer.

[0033] The shared semantic representation is input into a deep neural network classifier, which outputs the final communication intent category and confidence score.

[0034] The loss function for the collaborative training includes classification loss and view loss. Figure 1 The weighted sum of consistency regularization terms, where the view Figure 1 Consistency regularization terms constrain the distance between different view representations in the feature space.

[0035] Furthermore, it also includes:

[0036] Analyze key turning points in the evolution of intent using a temporal attention model to generate time-stamped sequences;

[0037] A temporal attention model is introduced into the deep classifier to calculate the contribution of each time step to the final classification decision;

[0038] The attention weights are obtained by softmax normalization of the attention scores, where the attention scores are obtained by scoring the elements in the encoder's hidden state sequence using a fully connected layer;

[0039] Based on attention weights, key turning points in the intention evolution process are identified, and the time points when the attention weights exceed the threshold are marked as key turning points, generating a time-stamped sequence of intention changes.

[0040] Furthermore, it also includes:

[0041] Discriminative features are extracted through interpretability analysis to generate a communication intent identification report;

[0042] The SHAP method is used to analyze the contribution of each feature to the classification results, calculate the Shapley value for each feature dimension, and select the N features with the highest absolute Shapley values ​​as the most discriminative feature combination.

[0043] Based on the extracted discriminative features and time-stamped sequences, a communication intent recognition report is generated, which includes information such as intent category, key behavioral features, and evolutionary trajectory.

[0044] Based on the identified communication intent categories, traffic control policy recommendations are generated using predefined mapping rules between intent categories and control policies.

[0045] Furthermore, the hierarchical topic model adopts a two-layer structure:

[0046] The first layer uses the latent Dirichlet assignment to represent the sequence of session state evolution as a probability distribution of behavioral topics. The topic assignment of each state is updated iteratively through Gibbs sampling, and the session-topic distribution parameters and topic-state distribution parameters are calculated.

[0047] The second layer uses a hierarchical Dirichlet process to automatically determine the number of intent categories and calculate the intent distribution. It uses the Chinese restaurant process to cluster behavioral topics and adaptively determines the number of intent categories through variational inference.

[0048] The posterior probability of each session belonging to each intent category is calculated using Bayesian inference. The category with the highest posterior probability is selected as the final communication intent category, and the posterior probability value is used as the confidence score.

[0049] This invention provides a system for intelligent analysis based on multi-source data, used to execute the aforementioned method for intelligent analysis based on multi-source data, comprising:

[0050] The metadata acquisition module is used to acquire multidimensional metadata of the encrypted communication stream and generate a set of feature vectors.

[0051] The state modeling module is used to calculate the state transition probability of a communication session using a Markov chain model and generate a session state evolution sequence.

[0052] The intent analysis module is used to analyze the session state evolution sequence based on a hierarchical topic model and output the communication intent category and confidence score.

[0053] The continuous representation module is used to map discrete states to a continuous semantic space using a variational autoencoder;

[0054] The relation analysis module is used to analyze the communication relation context through graph neural networks and generate relation embedding vectors.

[0055] The fusion classification module is used to learn and fuse multi-dimensional features based on multiple views, and output the final communication intent category.

[0056] The interpretability module is used to analyze key turning points in the evolution of intent, extract discriminative features, and generate a communication intent identification report.

[0057] The beneficial effects of this invention are as follows: it overcomes the limitation that static feature analysis cannot reflect the complete life cycle of a session, solves the semantic gap problem caused by the limited dimensions of metadata information, enhances the ability to identify spoofing behavior by introducing relational context through graph neural networks, and improves the accuracy and robustness of intent recognition by comprehensively considering behavioral trajectories, session features and relational background through multi-view fusion mechanism. Attached Figure Description

[0058] Figure 1 This is a flowchart of the method for intelligent analysis of multi-source data according to the present invention;

[0059] Figure 2 This is a heatmap of the state transition probability matrix of the present invention, showing the state transition probability matrix calculated by the Markov chain model in step 200;

[0060] Figure 3 This is a bar chart showing the probability distribution of communication intent categories in this invention, illustrating the probability distribution of each communication intent category output by the deep neural network classifier in step 302.

[0061] Figure 4 This is a line graph of the temporal attention weight distribution of the present invention, showing the contribution of each time window calculated by the temporal attention model in step 303 to the classification decision;

[0062] Figure 5 This is a horizontal bar chart ranking the discriminative features of the present invention, showing the discriminative features obtained by the SHAP method in step 304 and their contribution to the classification results. Detailed Implementation

[0063] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0064] This invention provides a method for intelligent analysis based on multi-source data, such as... Figure 1 As shown, it includes the following steps:

[0065] Step 100: Obtain multidimensional metadata of the encrypted communication stream and generate a set of feature vectors;

[0066] In this step, a multidimensional metadata sequence of the encrypted communication stream is obtained from the network traffic acquisition device, including packet length sequences, packet arrival time interval sequences, TLS handshake information, certificate chain data, and DNS query records. The obtained metadata is preprocessed and features are extracted to generate temporal feature vectors and statistical feature vectors.

[0067] The aforementioned data preprocessing includes: for non-numerical data, one-hot encoding of fields such as handshake type and cipher suite in TLS handshake information to convert them into numerical vectors; label encoding of fields such as certificate issuer and validity period in certificate chain data; and mapping of domain names in DNS query records to numerical identifiers using hash functions. For numerical data, features such as packet length and arrival time interval are scaled to the range of 0 to 1 using the min-max normalization method to eliminate the influence of different units on subsequent analysis.

[0068] Furthermore, the specific implementation method of the aforementioned one-hot encoding is as follows: For the TLS handshake type field, count all handshake types appearing in the training dataset, establish a mapping table from type to index, and for fields containing... Datasets of different handshake types, each handshake type mapped to A binary vector of dimension 1, where only the corresponding position is 1 and the rest are 0; for example, if the handshake type set is {ClientHello, ServerHello, Certificate}, then ClientHello is encoded as [1,0,0], ServerHello is encoded as [0,1,0], and Certificate is encoded as [0,0,1].

[0069] Furthermore, the specific implementation method of the aforementioned label encoding is as follows: For the certificate issuer field, all unique certificate issuers appearing in the training dataset are sorted in alphabetical order, and each issuer is assigned a consecutive integer label starting from 0; for example, if the dataset contains 3 different certificate issuers {CA_A, CA_B, CA_C}, they are encoded as 0, 1, and 2 respectively.

[0070] Furthermore, the aforementioned hash function employs the MurmurHash3 algorithm, mapping the DNS domain name string to a 32-bit unsigned integer. The specific calculation steps are as follows: the domain name string is converted into a byte sequence; the hash seed is initialized to a fixed value of 0x9747b28c; the byte sequence is divided into blocks, with each group of four bytes undergoing a mixing operation; finally, the hash value is avalanche mixing to obtain the final 32-bit integer identifier.

[0071] Furthermore, the calculation formula for the aforementioned minimum-maximum normalization method is as follows:

[0072] For the original eigenvalues Normalized value The calculation is as follows:

[0073]

[0074] in and These are the minimum and maximum values ​​of the feature in the training dataset, respectively.

[0075] when When, set To avoid division by zero errors.

[0076] After preprocessing, feature extraction is performed, including the temporal feature vector which includes the time series distribution of data packet length and the change pattern of arrival time interval;

[0077] Statistical feature vectors include statistical indicators such as average packet length, packet length variance, flow duration, and total number of packets.

[0078] Furthermore, each feature vector in the aforementioned feature vector set Dimensions Set it to 48.

[0079] Specifically, the temporal features include 32 dimensions:

[0080] The first 16 quantiles (16-dimensional) of the data packet length sequence and the first 16 quantiles (16-dimensional) of the arrival time interval sequence.

[0081] The statistical features include 16 dimensions: average packet length, packet length variance, packet length skewness, packet length kurtosis, flow duration, total number of packets, number of uplink packets, number of downlink packets, number of uplink bytes, number of downlink bytes, average arrival time interval, time interval variance, TLS handshake type encoding (2-dimensional), and cipher suite encoding (2-dimensional).

[0082] Feature vector dimension It can comprehensively characterize the multidimensional features of encrypted communication streams.

[0083] Step 200: Calculate the state transition probability of the communication session using the Markov chain model and generate the session state evolution sequence;

[0084] In this step, the feature vector set generated in step 100 is input into the Markov chain model to model the state transitions of the communication session. Specifically, the communication session is divided into time windows, and the state transition probability matrix within each time window is calculated. ,in Indicates from state Transition to state The probability of a state transition. The formula for calculating the state transition probability is:

[0085]

[0086] in, Indicates from state Transition to state Number of observations This indicates the total number of states.

[0087] Based on the calculated state transition probability matrix, a session state evolution sequence is generated:

[0088]

[0089] in Indicates time The session state.

[0090] Furthermore, the aforementioned state transition probability matrix The following constraints must be satisfied for any state:

[0091] ,have and That is, each state The sum of the probabilities of transitioning to all possible states is 1. When state When no transition has ever occurred in the training data (i.e.) ), setting status The transition probability is uniformly distributed. This ensures the integrity of the state transition probability matrix.

[0092] The input to the aforementioned Markov chain model is the set of feature vectors generated in step 100. ,in The output is the number of metadata records for the communication session; the output is the session state evolution sequence. ,in This refers to the number of time windows. The specific steps include:

[0093] First, divide the communication session into fixed time window sizes. The time window is divided into seconds, and the feature vectors within each time window are aggregated and statistically analyzed.

[0094] Secondly, since the aggregated feature vectors contain statistical indicators of different dimensions such as average packet length, packet length variance, flow duration, and total number of data packets, these feature vectors are standardized using the Z-score standardization method to eliminate the influence of differences in dimensions on the clustering results.

[0095] Then, K-means clustering is used to map the standardized feature vectors to... There are discrete states, where The value range is 5 to 10, and the optimal number of states is selected by the profile coefficient;

[0096] Next, the frequency of state transitions between adjacent time windows is counted. Construct the state transition frequency matrix;

[0097] Finally, the transition frequency matrix is ​​normalized to obtain the state transition probability matrix. Based on this matrix and the initial state distribution, a session state evolution sequence is generated.

[0098] Furthermore, the aforementioned Z-score standardization method is calculated as follows: for the feature vector... 3D eigenvalues Standardized value The calculation is as follows:

[0099]

[0100] in For the first The mean and standard deviation of the dimensional feature across all time windows are:

[0101]

[0102] when When, set To avoid division by zero errors.

[0103] Furthermore, the specific implementation steps of the aforementioned K-means clustering algorithm are as follows:

[0104] Step 1: Randomly select from the standardized feature vectors 1 sample was used as the initial cluster center.

[0105] ;

[0106] Step 2, for each feature vector Calculate its Euclidean distance to each cluster center:

[0107]

[0108] Will Assign to the nearest cluster center:

[0109]

[0110] Step 3: For each cluster, recalculate the cluster center as the mean of all samples within that cluster.

[0111]

[0112] in For the first Each cluster contains a set of samples;

[0113] Step 4: Repeat steps 2 and 3 until the cluster centers no longer change or the maximum number of iterations of 200 is reached;

[0114] Step 5: Use the cluster number to which the feature vector of each time window belongs as the status label of that time window.

[0115] Furthermore, the aforementioned time window size The second is determined based on the time scale of the handshake and data transmission phases of a typical encrypted communication protocol, and the size of the time window. A second can capture both the stable features within a state and the dynamic transitions between states.

[0116] Furthermore, the aforementioned method for calculating the silhouette coefficient is as follows: for each candidate state number

[0117] The K-means algorithm is used to cluster the standardized feature vectors to obtain the clustering results; for each sample Calculate the average distance between it and other samples in the same cluster. and the average distance to the nearest heterogeneous sample The silhouette coefficient of this sample is:

[0118]

[0119] The average silhouette coefficient of all samples is taken as the overall silhouette coefficient of the clustering result:

[0120]

[0121] in The total number of samples;

[0122] The selection makes the profile coefficient The maximum number of states is taken as the optimal number of states. .

[0123] Step 300: Analyze the session state evolution sequence based on the hierarchical topic model, and output the communication intent category and confidence score;

[0124] In this step, the session state evolution sequence generated in step 200 is input into the hierarchical topic model to infer the communication intent category behind each state. The hierarchical topic model adopts a two-layer structure: the first layer maps the session state evolution sequence to behavioral topics, and the second layer maps behavioral topics to intent categories. The posterior probability of each session belonging to each intent category is calculated using Bayesian inference, and the communication intent category and its corresponding confidence score are output.

[0125] It should be noted that the first layer of the hierarchical topic model uses Latent Dirichlet Allocation (LDA) to represent the sequence of session state evolution as a probability distribution of behavioral topics; the second layer uses Hierarchical Dirichlet Process (HDP) to automatically determine the number of intent categories and calculate the intent distribution.

[0126] The input to the aforementioned potential Dirichlet assignment is the session state evolution sequence:

[0127]

[0128] Each of them Indicates discrete state labels;

[0129] The output is the behavioral topic distribution of the session. ,in The preset number of topics.

[0130] The specific operation is as follows: treat the state sequence as a document, treat the state labels as words, use Gibbs sampling to iteratively update the topic assignment for each state, and calculate the session-topic distribution parameters. Theme-State Distribution Parameters The number of iterations is set to 1000, and the Dirichlet prior parameters are... and .

[0131] Furthermore, the specific implementation steps of the aforementioned Gibbs sampling are as follows:

[0132] Step 1, randomly initialize each state. Topic Allocation ;

[0133] Step 2, initialize the counting matrix, including session-topic counts. (conversation) Assigned to topic (Number of states) and topic-state count (theme medium state (number of occurrences)

[0134] Step 3, for each state First, remove it from the current topic and update the counting matrix:

[0135]

[0136]

[0137] Step 4, based on conditional probability:

[0138]

[0139] For state Resampling topic ,in Size of the state vocabulary;

[0140] Step 5, Update the counting matrix and ;

[0141] Step 6: Repeat steps 3 to 5 to traverse all states and complete one iteration;

[0142] Step 7: Repeat step 6 for a total of 1000 iterations;

[0143] Step 8: Calculate the session-topic distribution based on the final counting matrix. Theme-State Distribution .

[0144] Furthermore, the aforementioned LDA model parameters satisfy the following constraints:

[0145] Dirichlet Priors and Values and Ensure the sparsity of topic and state distribution;

[0146] Conversation-Topic Distribution satisfy and This is the K-dimensional probability simplex;

[0147] Topic-State Distribution Each line satisfy and ;

[0148] Conditional probability and ;

[0149] All elements in the counting matrix and It is a non-negative integer.

[0150] Furthermore, the number of the aforementioned topics Set to 20. Number of topics. The value was chosen based on the evaluation of the perplexity metric on the validation dataset:

[0151] In candidate values Within the specified range, train the LDA model and calculate the perplexity of the validation set, then select the model with the lowest perplexity. value;

[0152] Verification experiments showed that when The model achieves the lowest perplexity on the validation set and can effectively capture the diversity of behavioral themes in different communication sessions.

[0153] The input to the aforementioned hierarchical Dirichlet process is the behavioral topic distribution of the first layer's output. ;

[0154] The output includes intent category assignment and the number of intent categories. .

[0155] The specific operation is as follows: The Chinese Restaurant Process (CRP) is used to cluster behavioral topics; variational inference is used to adaptively determine the number of intent categories; the probability of each topic belonging to each intent category is calculated; and the lumped parameters are set to... .

[0156] Furthermore, the number of the aforementioned intent categories The number of classes is adaptively determined by the Hierarchical Dirichlet Process (HDP) and requires no pre-setting. During training, the HDP dynamically adjusts the number of classes based on the intrinsic structure of the data, through lumped parameters. Controlling the tendency to generate new categories. In practical applications, for encrypted communication datasets in enterprise network environments, the number of intent categories learned through HDP is... Typically, the range is between 8 and 15, covering normal business intents (such as file transfer, video conferencing, cloud service access, email communication, etc.), suspicious behavior intents (such as abnormal traffic patterns, unusual communication periods, etc.), and malicious behavior intents (such as data leakage, remote control, covert channel communication, etc.).

[0157] The input for the aforementioned Bayesian inference is the behavioral topic distribution of the conversation. And topic-intent category mapping probability;

[0158] The output is the posterior probability that the session belongs to each intent category. ,in Indicates the category of intent.

[0159] The specific steps are as follows: According to Bayes' theorem:

[0160]

[0161] Calculate the posterior probability, where the prior probability is... Likelihood probability is obtained from historical data statistics. The category with the highest posterior probability is selected as the final communication intent category by the joint calculation of the outputs of LDA and HDP, and the posterior probability value is used as the confidence score.

[0162] In this embodiment of the application, in order to improve the accuracy of intent recognition, the following steps are included in addition to step 200:

[0163] Step 201: Use a variational autoencoder to map the discrete state to a continuous semantic space to generate a continuous representation of the intent trajectory;

[0164] The session state evolution sequence output from step 200 is input into a variational autoencoder (VAE) to learn the representation of the session state evolution sequence in a continuous semantic space. The variational autoencoder consists of two parts: an encoder and a decoder. The encoder processes the session state evolution sequence... Mapping to latent variables Distribution The decoder reconstructs the session state evolution sequence from latent variables. By optimizing the variational lower bound (ELBO), it learns semantic representation vectors for different intent categories, generating continuous representations of intent trajectories. .

[0165] The aforementioned variational autoencoder consists of an embedding layer, two LSTM layers, and two fully connected layers. The embedding layer sequences the session state evolution:

[0166]

[0167] Mapped to an embedded vector sequence:

[0168]

[0169] in , For the embedded dimension.

[0170] The first LSTM layer encodes the embedding vector sequence and outputs the hidden state sequence:

[0171]

[0172] Each hidden state The calculation formula is:

[0173]

[0174]

[0175]

[0176]

[0177]

[0178]

[0179] in, This represents the sigmoid activation function. This represents element-wise multiplication. , , , This is the weight matrix. , , , This is the bias vector. The second LSTM layer further encodes the output of the first layer, and its calculation process is the same as that of the first layer, outputting a sequence of hidden states. The hidden state at the last moment. Input the two fully connected layers respectively to generate the mean vector of the latent variables. Sum of logarithmic variance vector :

[0180]

[0181]

[0182] in, , This is the weight matrix. , This is the bias vector. Latent variables are sampled using reparameterization techniques. :

[0183]

[0184] The decoder of the aforementioned variational autoencoder consists of a fully connected layer and two LSTM layers. The fully connected layer stores the latent variables... Mapped to the initial hidden state:

[0185]

[0186] The first LSTM layer with As the initial hidden state, at each time step Receive the output from the previous moment The embedding vector is used as input, and the hidden state is output. The second LSTM layer further decodes the output of the first layer, outputting a sequence of hidden states:

[0187]

[0188] Each hidden state The probability distribution mapped to the reconstructed state through the softmax layer is as follows:

[0189]

[0190] in This is the weight matrix. This is the bias vector.

[0191] The input to the aforementioned variational autoencoder is a sequence of session state evolutions:

[0192]

[0193] in The time calculated in step 200 The session state;

[0194] The output includes implicit variables. And the hidden state sequence of the second layer LSTM of the encoder:

[0195]

[0196] Among them, latent variables The hidden state sequence represents the global representation of the entire session state evolution sequence in a continuous semantic space. This represents the local representation at each time step, used for subsequent temporal attention analysis. The variational autoencoder employs an unsupervised learning training mode, learning continuous representations of the session state evolution sequence by maximizing the variational lower bound (ELBO), without needing intent category labels. The optimization strategy is the Adam optimizer, with a learning rate set to... The loss function is the negative of the variational lower bound (ELBO):

[0197] Furthermore, the parameter initialization method for the aforementioned variational autoencoder is as follows: all weight matrices are initialized using Xavier uniform initialization, i.e., sampled from a uniform distribution:

[0198]

[0199] in For input dimensions, For output dimensions;

[0200] All bias vectors are initialized to zero vectors;

[0201] The forget gate bias of the LSTM is initialized to 1.0 to facilitate long-term dependency learning.

[0202] Furthermore, the training convergence condition for the aforementioned variational autoencoder is as follows: ELBO loss is calculated on the validation dataset, and the convergence occurs when the improvement in validation loss is less than a threshold over 10 consecutive training epochs. When the model converges, training is stopped; or training is stopped when the maximum number of training epochs (100) is reached. The training batch size is set to 32, and all training samples are traversed once in each training epoch.

[0203] Furthermore, the aforementioned embedding dimension Set to 128, LSTM hidden layer dimension Set to 256, hidden variable dimension Set to 64. Embedding dimension LSTM hidden layer dimension and latent variable dimensions The settings are based on the following considerations:

[0204] Embedded Dimension It can fully represent the semantic information of discrete states;

[0205] LSTM hidden layer dimension It can capture long-term dependencies in the session state evolution sequence;

[0206] Latent variable dimensions Avoid overfitting while maintaining representational power.

[0207]

[0208] The first term is the reconstruction loss, calculated as cross-entropy:

[0209]

[0210] The second term is the KL divergence regularization term:

[0211]

[0212] in, This represents the prior distribution of the latent variable.

[0213] Furthermore, the definitions and value ranges of each parameter in the aforementioned VAE loss function are as follows:

[0214] The mean vector of latent variables:

[0215]

[0216] Unconstrained, range of values ​​is ;

[0217] Logarithm of the variance vector:

[0218]

[0219] There are no constraints, and the actual variance is:

[0220]

[0221] Always positive;

[0222] Reconstruction probability:

[0223]

[0224] Guaranteed by the softmax function;

[0225] KL divergence:

[0226]

[0227] According to the properties of information theory, it is always non-negative;

[0228] Total loss function of variational autoencoder The range of values ​​is The training objective is to minimize the total loss function of the variational autoencoder.

[0229] In this embodiment of the application, in order to enhance the ability to identify camouflage behavior, the following steps are also included in addition to step 300:

[0230] Step 301: Analyze the communication relationship context using a graph neural network to generate relationship embedding vectors;

[0231] Construct a communication graph between source and destination IP pairs, where nodes represent IP addresses and edges represent historical communication relationships. Communication frequency, duration, and data volume are used as attributes of the edges. A graph neural network is used to learn the representations of nodes and edges, and a relationship embedding vector for each IP pair is generated by aggregating neighbor node information. .

[0232] The aforementioned graph neural network employs a graph attention network architecture, comprising an input layer, three graph attention layers, and an output layer. The input layer receives the communication relationship graph. ,in For a set of nodes (IP addresses). This is a set of edges (communication relationships). Each node... initial feature vector By node The historical communication statistics for each IP address include communication frequency, average traffic, and number of connections; each edge eigenvectors From source IP With target IP The features are composed of attributes such as communication frequency, duration, and data volume. Since node features and edge features contain attributes with different units (communication frequency in times per second, data volume in bytes, duration in seconds, etc.), the node feature vector and edge feature vector are scaled to the range of 0 to 1 using the min-max normalization method to eliminate the impact of unit differences on graph neural network learning.

[0233] The first graph attention layer of the aforementioned graph neural network applies attention to each node. Calculate its relationship with neighboring nodes Attention coefficient between :

[0234]

[0235]

[0236] in, The node feature transformation matrix, Let be the edge feature transformation matrix. For attention weight vectors, This represents a vector concatenation operation. Represents a node The set of neighboring nodes. Based on the attention coefficient, the node... First layer output features We obtain the following by weighted aggregation of neighbor node features:

[0237]

[0238] in, The activation function is ELU. To stabilize the training process, multi-head attention is used, and the calculation... The independent attention coefficients and aggregation results are concatenated and then subjected to a linear transformation to obtain the final first-layer output:

[0239]

[0240] in, This represents the number of attention heads. The calculation process for the attention layers in the second and third graphs is the same as that in the first layer, respectively using... and As input, output and The output layer performs average pooling on the output of the third layer, merging the results of multi-head attention into a single representation.

[0241]

[0242] For each IP pair Embed the source node Embedding the target node Concatenate the vectors to obtain the relation embedding vector:

[0243]

[0244] The input to the aforementioned graph neural network is a communication relationship graph. Including node feature matrix Sum of edge feature matrices The output is the relation embedding vector for each IP pair. ,in This represents the output dimension of the third graph attention layer. The graph neural network employs a semi-supervised learning training mode, using IP pairs labeled with communication intent categories as training samples. The optimization strategy is the Adam optimizer, and the learning rate is set to... The weight decay coefficient is set to The loss function is the cross-entropy loss for node classification:

[0245] Furthermore, the aforementioned parameter initialization method for the graph neural network is as follows: node feature transformation matrix Sum of edge feature transformation matrix Glorot initialization is used, which means sampling from a uniform distribution:

[0246]

[0247] Attention weight vector Using normal distribution initialization;

[0248] Classification layer weight matrix Initialize using Xavier, bias vector Initialize to a zero vector.

[0249] Furthermore, the convergence condition for training the aforementioned graph neural network is:

[0250] The classification accuracy is calculated on the validation dataset. When the improvement in validation accuracy is less than 0.001 over 15 consecutive training epochs, the model is considered to have converged and training is stopped.

[0251] Alternatively, training can be stopped when the maximum number of training cycles (200) is reached.

[0252] The training batch size was set to 128, and a neighbor sampling strategy was adopted, with each node sampling a maximum of 20 neighbor nodes to control computational complexity.

[0253] Furthermore, the aforementioned initial feature dimension of the nodes Set to 32, edge feature dimension Set to 16, the output dimension of the first graph attention layer. Set to 64, the output dimension of the second layer. Set to 128, the output dimension of the third layer. Set to 128, attention head count Set to 8. Initial feature dimension of the node. Edge feature dimension Output dimensions and number of attention heads for each attention layer in the graph. The settings are based on the following considerations: initial feature dimensions of nodes and edge feature dimension It can fully represent historical communication statistics of IP addresses and the relationship attributes between IP pairs; the output dimension of the graph attention layer increases layer by layer ( , , This allows the network to learn more abstract topological features layer by layer; the number of attention heads It can capture different relational patterns from multiple subspaces, thereby improving the model's representational power.

[0254]

[0255] in, For the set of labeled IP pairs, The total number of communication intent categories, For IP pairs The true label (one-hot encoding). IP pairs predicted by the model Category The probability is obtained through relation embedding vectors. After the softmax classification layer, the following is obtained:

[0256]

[0257] in This is the classification weight matrix. This is the bias vector.

[0258] Furthermore, the parameters in the aforementioned graph neural network loss function satisfy the following constraints:

[0259] Authentic Labels and That is, for each IP pair There is exactly one category with a label of 1, and the rest with a label of 0;

[0260] Predicted probability and By the softmax function:

[0261]

[0262] Guarantee, among which For the first The class's logit value;

[0263] Cross-entropy loss ,when The minimum value is 0.

[0264] IP pair set already labeled Size Should meet Ensure that there are at least 10 training samples for each intent category.

[0265] Step 302: Based on multi-view learning, fuse multi-dimensional features and output the final communication intent category;

[0266] The continuous representation of the intent trajectory generated in step 201, the session feature vector generated in step 100, and the relationship embedding vector generated in step 301 are used as three views. Through co-training, the representations of different views are aligned in a shared semantic space. The fused feature vector is input into a deep neural network classifier, which outputs the final communication intent category and confidence score.

[0267] The aforementioned collaborative training includes the following steps:

[0268] Step 1: Initialize the parameters of three view-specific encoders and one shared mapping layer;

[0269] Step 2: Randomly sample a batch of samples from the training dataset and obtain three view data for each sample;

[0270] Step 3: Input the three view data into the corresponding view-specific encoders to obtain their respective view representations. , , ;

[0271] Step 4: Calculate the consistency loss between views and constrain the distance between the representations of different views in the feature space;

[0272] Step 5: Concatenate the three view representations and input them into the shared mapping layer to obtain the shared semantic representation. ;

[0273] Step 6: Input the shared semantic representation into the classifier and calculate the classification loss;

[0274] Step 7: Weight the classification loss and consistency loss to obtain the total loss;

[0275] Step 8: Update the parameters of all encoders, shared mapping layers, and classifiers using gradient descent;

[0276] Step 9: Repeat steps 2 through 8 until the preset number of iterations is reached or the loss converges.

[0277] Through the above collaborative training process, the encoders of the three views cooperate with each other, maintaining the discriminative information of their respective views while aligning in the shared space, thereby achieving effective fusion of multi-source information.

[0278] The aforementioned multi-view learning comprises two parts: a view-specific encoder and a shared semantic space mapping. For the intent trajectory view, a two-layer fully connected network is used as the encoder to map latent variables. Mapping to view representation :

[0279]

[0280] in, , This is the weight matrix. , For bias vectors, This represents the hidden layer dimension. For the session feature view, a two-layer fully connected network is also used to process the feature vectors. Mapping to view representation :

[0281]

[0282] For relation embedding views, a two-layer fully connected network is used to embed the relation vectors. Mapping to view representation :

[0283]

[0284] The three view representations are concatenated and then projected onto a shared semantic space through a shared mapping layer:

[0285]

[0286] in, To share the mapping weight matrix, For bias vectors, For shared semantic representation.

[0287] The aforementioned deep neural network classifier employs a multi-layer perceptron architecture, comprising an input layer, three hidden layers, and an output layer. The input layer receives a shared semantic representation. The first hidden layer is computed using a fully connected transformation and ReLU activation:

[0288]

[0289] in, This is the weight matrix. For bias vectors, Let be the dimension of the first hidden layer. To prevent overfitting, a dropout operation is applied after the first hidden layer, with probability... Randomly discard some neurons:

[0290]

[0291] The second hidden layer pair Further transformations are needed:

[0292]

[0293] in, This is the weight matrix. This is the bias vector. Applying the dropout operation similarly yields:

[0294]

[0295] The calculation for the third hidden layer is as follows:

[0296]

[0297] The output layer will output the third hidden layer. The probability distribution mapped to communication intent categories using the softmax function:

[0298]

[0299] in, This is the output layer weight matrix. For bias vectors, For the predicted intent category probability distribution, This represents the total number of intent categories.

[0300] The input to the aforementioned deep neural network classifier is a shared semantic representation. The output is the probability distribution of the communication intent category. The category with the highest probability value in the probability distribution is taken as the final communication intent category, that is:

[0301]

[0302] Corresponding probability value As a confidence score.

[0303] The classifier is trained using supervised learning, with the Adam optimizer as the optimization strategy and a learning rate set to [value missing]. The batch size is set to 64.

[0304] The loss function is the cross-entropy loss and the view loss. Figure 1 Weighted sum of consistency regularization terms:

[0305] Furthermore, the parameter initialization method for the aforementioned multi-view learning model is as follows: all weight matrices of the view-specific encoder and classifier are initialized using He, i.e., from a normal distribution. Mid-sampling, of which For the input dimension, the He initialization method is suitable for the ReLU activation function; all bias vectors are initialized to zero vectors; the mapping layer weight matrix is ​​shared. Orthogonal initialization is used to ensure linear independence between different view representations.

[0306] Furthermore, the convergence condition for the aforementioned multi-view learning model is as follows: Classification accuracy is calculated on the validation dataset. If the improvement in validation accuracy is less than 0.0005 over 20 consecutive training epochs, the model is considered converged and training is stopped; or training is stopped when the maximum number of training epochs (150) is reached. A learning rate decay strategy is adopted: if the validation accuracy does not improve for 5 consecutive epochs, the learning rate is decayed to 0.5 times its original value. The minimum learning rate is set to... .

[0307] Furthermore, the dimension parameter in the aforementioned multi-view learning is set to: the hidden layer dimension of the view-specific encoder. View represents dimension Shared semantic space dimension The first hidden layer dimension of the classifier Second hidden layer dimension The third hidden layer dimension In multi-view learning, the hidden layer dimension of the view-specific encoder. View representation dimensions Shared semantic space dimension The settings for the hidden layer dimensions of each classifier are based on the following considerations: view representation dimensions. Capable of retaining key discriminative information from each view; sharing semantic space dimensions. The dimensions of the three view representations are spliced ​​together to ensure the complete integration of information from different views; the dimensions of the hidden layer of the classifier decrease layer by layer (256 → 128 → 64), forming a funnel-shaped structure, which is beneficial for the layer-by-layer abstraction of features and classification decisions.

[0308]

[0309] Among them, cross-entropy loss The calculation is as follows:

[0310]

[0311] in, The number of training samples. For the sample The true label (one-hot encoding). To predict probabilities. (See) Figure 1 Consistency regularization term Constrain different views to have similar representations in a shared semantic space:

[0312]

[0313] in, This is the regularization coefficient.

[0314] Furthermore, the parameters in the aforementioned multi-view learning loss function satisfy the following constraints:

[0315] Regularization coefficient Values Balancing classification accuracy and visual accuracy Figure 1 To the point of being compatible;

[0316] View representation Unconstrained, range of values ​​is ;

[0317] See Figure 1 Sexual damage Guaranteed by the nonnegativity of the square of the Euclidean distance, when The minimum value is 0.

[0318] Cross-entropy loss ;

[0319] Total loss function ;

[0320] Authentic Labels and ;

[0321] Predicted probability and ;

[0322] Number of training samples Should meet To ensure the effectiveness of model training.

[0323] In this embodiment of the application, in order to provide interpretable analysis results, the following steps are further included in addition to step 302:

[0324] Step 303: Analyze the key turning points in the evolution of intent using a temporal attention model to generate a time-stamped sequence;

[0325] A temporal attention model is introduced into the deep classifier to calculate the contribution of each time step to the final classification decision. Attention weights are then used. The calculation formula is:

[0326]

[0327] in, It is a moment The attention score is obtained by applying the fully connected layer to the encoder hidden state sequence output in step 201. Scoring each element yields:

[0328]

[0329] in For a moment The encoder's second-layer LSTM hidden state, Here is the attention weight matrix. For attention vectors, For bias vectors, It is the sequence length.

[0330] Identify key turning points in the intent evolution process based on attention weights, and assign attention weights that exceed a threshold. The time points are marked as key turning points to generate a time-stamped sequence of intended changes. .

[0331] Furthermore, the aforementioned temporal attention weights satisfy the following constraints:

[0332] Attention weight and This is guaranteed by softmax normalization;

[0333] Attention Score Unconstrained, the output range of the tanh activation function Determined by linear transformation;

[0334] Key inflection point threshold Values Ensure that the number of selected time points is appropriate;

[0335] Time stamped sequence Size ,satisfy ;

[0336] Sequence length Consistent with the number of time windows, the value range is... The duration of the communication session and the size of the time window are determined by the duration of the communication session. Decision made in seconds.

[0337] Furthermore, the parameters of the aforementioned temporal attention model , and This is achieved through end-to-end training. Specifically, a temporal attention layer is integrated into the classifier in step 302, sharing semantic representations. Based on this, an attention mechanism is used to process the encoder's hidden state sequence. Weighted aggregation is performed to obtain attention-enhanced representations. ,Will and The concatenated data is then input into the classifier. The attention parameters and classifier parameters are trained simultaneously, including the attention weight matrix. and attention vector Initialize using Xavier, bias vector Initialize to a zero vector.

[0338] Furthermore, the aforementioned attention weight threshold The method for determining the attention weights is as follows: statistically analyze the distribution of attention weights across all time steps on the validation dataset, and calculate the mean of the attention weights. and standard deviation Set the threshold to , where the coefficient Through grid search Choose the value within the range that maximizes the accuracy of key inflection point identification; experiments on the validation dataset show that when The recognition effect is optimal at this time, and the corresponding threshold is approximately 0.1. Attention weight threshold. It can effectively select time steps that contribute significantly to classification decisions and filter out noise interference.

[0339] Step 304: Extract discriminative features through interpretability analysis and generate a communication intent identification report;

[0340] The SHAP (SHapley Additive exPlanations) method is used to analyze the contribution of each feature to the classification results, and the most discriminative feature combinations are extracted. Based on the extracted discriminative features and time-stamped sequences, a communication intent recognition report containing information such as intent category, key behavioral features, and evolutionary trajectory is generated, and corresponding traffic control strategy suggestions are output.

[0341] The aforementioned SHAP method takes a trained deep neural network classifier model and the sample feature vectors to be interpreted as input. And a reference dataset; the output is the contribution value of each feature dimension to the classification result:

[0342]

[0343] in Indicates the first The Shapley values ​​for each feature are calculated. Specifically, a sample from the reference dataset is randomly selected as a baseline. The feature difference between the current sample and the baseline sample is calculated. For each feature dimension, the marginal contribution of that feature is calculated by enumerating all possible feature subset combinations. Monte Carlo sampling is used to approximate the Shapley value to reduce computational complexity, and the number of samplings is set to 100. The top-ranked Shapley values ​​by absolute value are selected. The features are considered as the most discriminative feature combinations.

[0344] Furthermore, the specific calculation method for the aforementioned Shapley value is as follows:

[0345] For features Its Shapley value Defined as the average marginal contribution of this feature across all possible combinations of feature subsets, the calculation formula is:

[0346] in For the set of all features, For features not included Feature subset, Indicates using only a subset of features The predicted output of the model. Since the computational complexity of completely enumerating all feature subsets is O(n log n), the actual computational complexity is O(n log n). Monte Carlo sampling is used for approximate calculation: 100 feature subsets are randomly sampled, and for each subset... Calculate the marginal contribution:

[0347]

[0348] The Shapley value is approximately:

[0349]

[0350] in For the first The feature subset of the subsample.

[0351] Furthermore, the aforementioned feature subset sampling strategy is as follows: for each sampling, each feature (except for feature ) is independently determined with a probability of 0.5. Whether the feature subset is included in the subset ensures that the size of the sampled feature subset follows a binomial distribution, covering all cases from the empty set to the entire set; for features included... extended subset The feature values ​​at the corresponding positions are set to the values ​​of the current samples. For features not included in the dataset, their values ​​are set to the average value of the reference dataset. This process constructs a partial feature input vector, which is then used by a classifier to obtain the predicted output. .

[0352] Furthermore, the number of the aforementioned discriminative features The selection method is as follows: Settings:

[0353]

[0354] That is, taking 10 and sharing semantic space dimensions. The smaller of 10% is rounded down. hour:

[0355]

[0356] therefore ;like Smaller (e.g.) If the number of features is too high, then select the number of features proportionally to avoid selecting too many redundant features. Number of discriminative features The selection strategy ensures the discriminative power of features while avoiding the problem of not being able to select a sufficient number of features when the feature dimensions are insufficient.

[0357] The aforementioned process for generating traffic control policy recommendations is as follows: Based on the communication intent categories output in step 302, specific policy recommendations are generated using predefined mapping rules between intent categories and control policies. Specifically, a mapping table is established between intent categories and control actions. Normal business intents (such as encrypted file transfer, video conferencing, cloud service access, etc.) are mapped to allow-pass policies; suspicious behavior intents (such as abnormal data transmission patterns, irregular communication periods, etc.) are mapped to rate limiting or alarm policies; and malicious behavior intents (such as data leakage, remote control, covert channel communication, etc.) are mapped to blocking or isolation policies. Each policy includes a control action type, priority, confidence score, and execution parameters. The confidence score directly uses the classification confidence score output in step 302, and the execution parameters are determined based on the specific values ​​of the discriminative features, such as setting a rate limiting threshold based on traffic volume or setting an alarm time window based on a time stamp sequence.

[0358] The following is an example of an application of the present invention, such as Figure 2-5 As shown, the implementation process is as follows:

[0359] Metadata for this encrypted communication session was obtained from a network traffic acquisition device. The monitoring period was 60 seconds, and a total of 127 data packets were captured. Some of the extracted raw metadata is shown in Table 1.

[0360] Table 1. Raw metadata of encrypted communication sessions (partial)

[0361]

[0362] After preprocessing and feature extraction of the original metadata, a 48-dimensional feature vector is generated. Some of the extracted features are shown in Table 2.

[0363] Table 2. Extracted feature vectors (partial dimensions)

[0364]

[0365] The communication session was divided into 12 time windows, each lasting 5 seconds. After aggregating and statistically standardizing the feature vectors within each time window using Z-score, K-means clustering was used to map them to 7 discrete states. The optimal number of states was determined by silhouette coefficient evaluation. The generated session state evolution sequence is as follows:

[0366] .

[0367] The calculated state transition probability matrix (partial) is shown in Table 3.

[0368] Table 3. State transition probability matrix (partial)

[0369]

[0370] The state transition probability matrix shows that the session exhibits a clear unidirectional evolution characteristic (state 1 → state 2 → state 3 → state 4 → state 5 → state 6), without the periodic state switching pattern commonly seen in normal business communication.

[0371] The session state evolution sequence is input into a hierarchical topic model. The first-layer LDA model outputs a 20-dimensional probability vector of behavioral topics for the session, with topics 7, 12, and 18 having relatively high probabilities of 0.342, 0.281, and 0.237, respectively. The second-layer HDP model maps behavioral topics to intent categories and calculates posterior probabilities through Bayesian inference.

[0372] After the variational autoencoder generates a continuous representation of the intent trajectory, the graph neural network generates a relation embedding vector, and after the multi-view learning fuses the features of the three views, the probability distribution of the communication intent category output by the deep neural network classifier is shown in Table 4.

[0373] Table 4 Probability Distribution of Communication Intent Categories

[0374]

[0375] The classifier identified the communication intent of this session as "data leakage" with a confidence score of 0.587.

[0376] The temporal attention model calculates the contribution of each time window to the classification decision, and the attention weight distribution is shown in Table 5.

[0377] Table 5 Temporal attention weight distribution

[0378]

[0379] Four key turning points (time windows 4, 5, 6, and 7) were identified, corresponding to a time period of 15-35 seconds. During this period, the session was in state 3, characterized by continuous high-volume uplink data transmission, which is key evidence for determining the intent to leak data.

[0380] The contribution of each feature dimension of the 192-dimensional shared semantic representation to the classification results was analyzed using the SHAP method. The top 10 discriminative features with the highest absolute Shapley values ​​are shown in Table 6.

[0381] Table 6 Discriminative features and their contribution

[0382]

[0383] The communication intent identification report generated based on the identification results is shown in Table 7.

[0384] Table 7 Communication Intent Identification Report

[0385]

[0386] Based on the report, the Security Operations Center immediately implemented a blocking strategy. Subsequent manual auditing confirmed that the server had indeed suffered an APT attack, and the attackers were sending sensitive data back to an external server. This implementation method accurately identified the data breach intent without decrypting the communication content, effectively preventing the data breach from occurring.

[0387] It is understood that data preprocessing methods known to those skilled in the art include data cleaning, data transformation, and data reduction. Data transformation includes type conversion and normalization and standardization. Although the dimensions and types of data were omitted in the description of the preceding embodiments, data preprocessing is a technical knowledge known to those skilled in the art and a prerequisite step in data processing. Therefore, the previously described well-known data preprocessing steps were not described independently.

[0388] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments based on the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method based on intelligent analysis of multi-source data, characterized in that, Includes the following steps: Obtain multidimensional metadata of the encrypted communication stream and generate a set of session feature vectors; The state transition probability of a communication session is calculated using a Markov chain model. The communication session is divided into time windows and mapped to discrete states to generate a session state evolution sequence. Based on the hierarchical topic model, the session state evolution sequence is analyzed. The session state evolution sequence is mapped to behavioral topics through latent Dirichlet assignment, and the behavioral topics are mapped to intent categories through the hierarchical Dirichlet process. The communication intent category and confidence score are output. A variational autoencoder is used to map discrete states to a continuous semantic space, generating a continuous representation of the intent trajectory; Graph neural networks are used to analyze the communication context and generate relationship embedding vectors. Based on multi-view learning and fusion of multi-dimensional features, the final communication intent category is output; The continuous representation of intent trajectory, session feature vector, and relation embedding vector are used as three views. Through collaborative training, the representations of different views are aligned in a shared semantic space. The three views are mapped to view representations using a view-specific encoder and projected onto a shared semantic space through a shared mapping layer. The shared semantic representation is input into a deep neural network classifier, which outputs the final communication intent category and confidence score. The state transition probability is calculated as follows: the probability of transitioning from state i to state j is equal to the number of observations of transitioning from state i to state j divided by the sum of the number of observations of transitioning from state i to all states.

2. The method based on intelligent analysis of multi-source data according to claim 1, characterized in that, The step of obtaining multi-dimensional metadata of the encrypted communication stream and generating a session feature vector set specifically includes: Obtain multidimensional metadata sequences of encrypted communication streams from network traffic acquisition devices, including packet length sequences, packet arrival time interval sequences, TLS handshake information, certificate chain data, and DNS query records; The acquired metadata is preprocessed, including: one-hot encoding of the handshake type and cipher suite fields in the TLS handshake information; tag encoding of the certificate issuer and validity period fields in the certificate chain data; mapping the domain name in the DNS query record to a numerical identifier using a hash function; and scaling the packet length and arrival time interval to the range of 0 to 1 using a minimum-maximum normalization method. Feature extraction is performed to generate temporal feature vectors and statistical feature vectors. The temporal feature vectors include the time series distribution of packet length and the change pattern of arrival time intervals, while the statistical feature vectors include the average packet length, packet length variance, flow duration, and total number of packets.

3. The method based on intelligent analysis of multi-source data according to claim 1, characterized in that, The step of using a Markov chain model to calculate the state transition probability of a communication session and generate a session state evolution sequence specifically includes: The communication session is divided into fixed time windows, and the feature vectors within each time window are aggregated and statistically analyzed. The aggregated feature vectors are standardized using the Z-score standardization method. The standardized value is equal to the original value minus the mean and then divided by the standard deviation. The standardized feature vectors are clustered using K-means to map them to n discrete states, and the optimal number of states is selected by the silhouette coefficient. Count the frequency of state transitions between adjacent time windows and construct a state transition frequency matrix; The transition frequency matrix is ​​normalized to obtain the state transition probability matrix. Based on this matrix and the initial state distribution, a session state evolution sequence is generated.

4. The method based on intelligent analysis of multi-source data according to claim 1, characterized in that, Also includes: The variational autoencoder includes an encoder and a decoder. The encoder maps the session state evolution sequence to the distribution of latent variables, and the decoder reconstructs the session state evolution sequence from the latent variables. The semantic representation vectors for different intent categories are learned by optimizing the variational lower bound, which includes a reconstruction loss and a KL divergence regularization term. The reconstruction loss is calculated as cross-entropy, and the KL divergence regularization term constrains the difference between the latent variable distribution and the prior distribution.

5. The method based on intelligent analysis of multi-source data according to claim 4, characterized in that, Also includes: Construct a communication relationship graph between source and destination IP pairs, where nodes represent IP addresses and edges represent historical communication relationships, with communication frequency, duration, and data volume as attributes of the edges; The graph attention network is used to learn the representations of nodes and edges, and the relation embedding vector of each IP pair is generated by aggregating the neighbor node information. The graph attention network calculates the attention coefficient between each node and its neighboring nodes, and obtains the output features of the node by weighted aggregation of the neighboring node features based on the attention coefficient.

6. The method based on intelligent analysis of multi-source data according to claim 5, characterized in that, Also includes: The loss function for the collaborative training includes a weighted sum of classification loss and a view consistency regularization term, where the view consistency regularization term constrains the distance between different view representations in the feature space.

7. The method based on intelligent analysis of multi-source data according to claim 6, characterized in that, Also includes: Analyze key turning points in intent evolution using a temporal attention model to generate time-stamped sequences; A temporal attention model is introduced into the deep classifier to calculate the contribution of each time step to the final classification decision; Attention weights are obtained by softmax normalization of attention scores, where attention scores are obtained by scoring elements in the encoder's hidden state sequence using a fully connected layer; Based on attention weights, key turning points in the intention evolution process are identified, and the time points when the attention weights exceed the threshold are marked as key turning points, generating a time-stamped sequence of intention changes.

8. The method based on intelligent analysis of multi-source data according to claim 7, characterized in that, Also includes: Discriminative features are extracted through interpretability analysis to generate a communication intent identification report; The SHAP method is used to analyze the contribution of each feature to the classification results, calculate the Shapley value for each feature dimension, and select the N features with the highest absolute Shapley values ​​as the most discriminative feature combination. Based on the extracted discriminative features and time-stamped sequences, a communication intent identification report containing intent category, key behavioral features, and evolutionary trajectory information is generated; Based on the identified communication intent categories, traffic control policy recommendations are generated using predefined mapping rules between intent categories and control policies.

9. The method based on intelligent analysis of multi-source data according to any one of claims 1 to 8, characterized in that, The hierarchical topic model adopts a two-layer structure: The first layer uses the latent Dirichlet assignment to represent the session state evolution sequence as a probability distribution of behavioral topics. The topic assignment of each state is updated iteratively through Gibbs sampling, and the session-topic distribution parameters and topic-state distribution parameters are calculated. The second layer uses a hierarchical Dirichlet process to automatically determine the number of intent categories and calculate the intent distribution. It uses the Chinese restaurant process to cluster behavioral topics and adaptively determines the number of intent categories through variational inference. The posterior probability of each session belonging to each intent category is calculated using Bayesian inference. The category with the highest posterior probability is selected as the final communication intent category, and the posterior probability value is used as the confidence score.

10. A system based on intelligent analysis of multi-source data, used to execute the method based on intelligent analysis of multi-source data as described in any one of claims 1-9, characterized in that, include: The metadata acquisition module is used to acquire multidimensional metadata of the encrypted communication stream. Generate a set of feature vectors; The state modeling module is used to calculate the state transition probability of a communication session using a Markov chain model and generate a session state evolution sequence. The intent analysis module is used to analyze the session state evolution sequence based on a hierarchical topic model and output the communication intent category and confidence score. The continuous representation module is used to map discrete states to a continuous semantic space using a variational autoencoder; The relation analysis module is used to analyze the communication relation context through graph neural networks and generate relation embedding vectors. The fusion classification module is used to learn and fuse multi-dimensional features based on multiple views, and output the final communication intent category. The interpretability module is used to analyze key turning points in the evolution of intent, extract discriminative features, and generate a communication intent identification report.

Images