A method for mining relations between multi-source heterogeneous data

By preprocessing and natural language processing of multi-source heterogeneous data, using phrase weights and improved Jaccard method, the problem of multi-source heterogeneous data relationship calculation in the existing technology is solved, and more accurate and comprehensive physical relationship mining is achieved. , improves the target positioning capabilities in the field of information security.

CN118761038BActive Publication Date: 2024-11-15DALIAN SHUCHEN TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411239546.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-15
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

The prior art is difficult to effectively use multi-source heterogeneous data for attack entity relationship calculations in the field of information security, resulting in the inability to accurately track and lock the target entity, and requires a large amount of training data and manual intervention, which has limitations in natural semantic processing.

Method used

Multi-source heterogeneous data is collected, pre-processed and natural language processing is performed, and entity similarity calculation is performed through phrase weight calculation and improved Jaccard method to realize the mining of multi-source heterogeneous data relationships.

Benefits of technology

It improves the scalability and generalization ability of inter-entity relationship mining, can position the target entity more comprehensively and accurately, ignores the limitations of entity attributes, and enhances the relationship mining ability of unstructured data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118761038B_ABST
    Figure CN118761038B_ABST
Patent Text Reader

Abstract

The invention discloses a method for mining multi-source heterogeneous data relations, S1: collecting multi-source heterogeneous data, including terminal data, transmission link data and network space data, and dividing the terminal data, transmission link data and network space data into structured data, semi-structured data and unstructured data; S2: pre-processing the structured data, semi-structured data and unstructured data respectively to obtain text data; S3: performing natural language processing on the pre-processed text data to obtain processed natural language; S4: performing word segmentation processing on the processed natural language to obtain entity data phrases; S5: determining phrase position weight, phrase part-of-speech weight and phrase keyword weight based on the entity data phrase, and obtaining phrase weight based on the phrase position weight, phrase part-of-speech weight and phrase keyword weight; S6: performing entity similarity calculation based on the phrase weight and the entity data phrase, thereby mining the data relations of the multi-source heterogeneous data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a method for mining multi-source heterogeneous data relations. Background Art

[0002] In today's digital and intelligent era, the convenient creation, processing and dissemination of various types of information have greatly facilitated our production and life, but also increased the risk of information security. With the rapid development of technology, traditional network attack methods are constantly updated and iterated, and new generation information security issues such as deep fakes, public opinion manipulation, cyber violence, privacy leakage, and IoT security are becoming increasingly prominent and frequently occurring, posing unprecedented challenges to individual information security and even social stability. From terminal devices to transmission links and cyberspace, attackers will inevitably leave traces of attacks when performing any form of network or system attacks, but generally all attackers' attack behaviors in the entire link are carried out anonymously, which greatly increases the difficulty of tracking and locking the real entity behind the target.

[0003] When tracking and locking the target, it is necessary to analyze whether the data belongs to the same entity. The main association methods currently include establishing Boolean model associations through specific identifiers, such as mobile phone numbers, network accounts, IP addresses, etc. However, in actual attack events, due to the attacker's intentional concealment of the identification features of the real identity involved, and various special circumstances that may easily cause the omission of the association between entities and ignore the value of the target entity behavior data, it may lead to the inability to accurately track and lock the target; and the probabilistic model association method generally uses technologies such as word2vec to convert the entity plain text data into a dense vector of fixed length through word segmentation, and through clustering algorithms such as vector space KNN, k-means and DBSCAN, similar phrases are clustered in the vector space to achieve clustering of attack entity information based on plain text. However, because the attack event-related data has the characteristics of multi-source heterogeneity, it is impossible to make good use of structured and unstructured data for relationship calculation to mine data relationships. In addition, this method requires a large amount of training data, labeled data, manual intervention, and has limitations in natural semantic processing. Summary of the invention

[0004] The present invention provides a method for mining multi-source heterogeneous data relationships to overcome the problems that the prior art may cause omission of inter-entity association relationships, ignore the value of target entity behavior data, and due to the multi-source heterogeneity of attack event-related data, there is a technical problem that structured and unstructured data cannot be well utilized for relationship calculation to mine data relationships, and a large amount of training data, labeled data, manual intervention is required, and there are limitations in natural semantic processing.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A method for mining multi-source heterogeneous data relationships, the specific steps of which include:

[0007] S1: Collect multi-source heterogeneous data, including terminal data, transmission link data and cyberspace data, and divide the terminal data, transmission link data and cyberspace data into structured data, semi-structured data and unstructured data;

[0008] S2: Preprocessing the structured data, semi-structured data and unstructured data respectively to obtain text data;

[0009] S3: performing natural language processing on the preprocessed text data to obtain processed natural language;

[0010] S4: Perform word segmentation on the processed natural language to obtain entity data phrases;

[0011] S5: determining a phrase position weight, a phrase part-of-speech weight, and a phrase keyword weight based on the entity data phrase, and obtaining a phrase weight based on the phrase position weight, the phrase part-of-speech weight, and the phrase keyword weight;

[0012] S6: Calculate entity similarity based on phrase weights and entity data phrases to mine data relationships among multi-source heterogeneous data.

[0013] Further, the phrase position weight includes entity identification weight, entity attribute weight, entity relationship weight, entity behavior weight and other weights;

[0014] The phrase part-of-speech weights include weights of nouns, verbs, adjectives, adverbs, conjunctions, prepositions, pronouns, interjections, and other non-core words.

[0015] Furthermore, the formula for calculating the weight of the phrase keyword is: ,

[0016] Among them, n i,j It is a phrase i In a piece of data related to the attack entity j The number of times it appears in k To traverse a piece of data j The index of all different phrases in ∑ k n k,j Is a piece of data related to the attack entity j The total number of occurrences of all phrases in the data j Total number of phrases;

[0017] ,in, is the total amount of data in the relevant attack entity data,j : t i ∈ d j Contains the phrase t i The number of data; d j Indicates the first j Item data;

[0018] Multiply TF and IDF to get the phrase keyword weight, that is, the value of TF-IDF:

[0019] The weight of the phrase keyword is normalized, and the normalization formula is:

[0020] in, t It is an attack entity related dataset d A phrase in ; TF-IDF( t , d ) is a phrase t In entity related datasets d TF-IDF value in; the denominator is the entity-related dataset d All phrases in t The square root of the sum of the squares of the TF-IDF values ​​of ′, that is, the entity-related dataset d The L2 norm of the TF-IDF vector;

[0021] Based on the phrase position weight, phrase part-of-speech weight and phrase keyword weight, the phrase weight is obtained as: weight = PositionWeight * POSWeight * TF-IDFWeight.

[0022] Furthermore, in S6, the process of calculating entity similarity based on phrase weight and entity data phrase is as follows:

[0023] Each entity data phrase set and its corresponding phrase weight are combined into a weighted vector, expressed as: ,

[0024] in, w n is the weight of each entity data phrase; a n is an entity data phrase;

[0025] The similarity calculation is based on the improved Jaccard method, and the formula is: ,

[0026] in, w iIt is a phrase i The feature weights of a i and b i They are the phrases in different entity data phrase sets. i The presence indicator of .

[0027] Furthermore, in S2, the structured data, semi-structured data and unstructured data are preprocessed respectively to obtain text data, which includes:

[0028] S21: Preprocessing the structured data and semi-structured data, the process is as follows:

[0029] Data deduplication: Identify and remove duplicates by comparing entire strings or calculating hash values ​​of text;

[0030] Missing value filling: fill missing values ​​and missed continuous data by using any of the methods including mean, median, mode, interpolation and model prediction;

[0031] Outlier processing: unify data units;

[0032] Eliminate noisy data: remove HTML tags, special characters and useless spaces;

[0033] Dimensionality reduction and normalization: Dimensionality reduction and normalization of static data based on Hash algorithm;

[0034] S22: The unstructured data includes text document data and multimedia data;

[0035] S221: The process of preprocessing the text document data includes:

[0036] Processing Excel files: Use Apache POI library to read and parse text information in Excel files;

[0037] Word file processing: Use Apache POI library to read and parse text information in Word documents;

[0038] Processing of CSV files: directly read through text file processing method;

[0039] Processing of PDF files: Use PDFBox to identify the text and layout in PDF files and extract the information therein; For PDF files containing non-text elements, use OCR technology to convert the text in the image into editable text format files, and then parse the editable text format files;

[0040] S222: The process of preprocessing the multimedia data includes:

[0041] Image processing: Process the image through the NeuralTalk tool to generate text information;

[0042] For speech processing: convert speech signals into text information through the DeepSpeech2 tool;

[0043] Video processing: extract channel information from the video and use DeepSpeech2 to convert the channel information into text information; use the frame difference method to extract the pictures whose picture changes exceed the preset threshold from the video, and perform picture semantic analysis to obtain text information.

[0044] Furthermore, in S3, the GPT model is used when performing natural language analysis on the preprocessed text data.

[0045] Furthermore, in S4, the HanLP word segmentation tool is used to perform word segmentation on the processed natural language.

[0046] Beneficial effects: The present invention collects all relevant multi-source heterogeneous data generated by the target attack event, and pre-processes the collected structured data, semi-structured data and unstructured data to obtain text data; natural language processing tools are used to perform natural language processing on the pre-processed text data to obtain processed natural language, thereby enhancing the relationship mining of unstructured data. The processed natural language is segmented using a word segmentation tool to obtain entity data phrases, and the phrase weights of the entity data phrases are determined. The entity similarity is calculated based on the phrase weights and the entity data phrases, thereby realizing multi-source heterogeneous data relationship mining; this method ignores the limited association relationships in the entity attributes, so that the full amount of entity data can participate in the relationship calculation, thereby being able to locate the target entity in a more comprehensive, accurate and targeted manner, solving the problem that the traditional relationship discovery method cannot effectively use multi-source heterogeneous data of unknown dimensions for deeper relationship discovery and mining, and greatly improving the scalability and generalization capabilities of entity relationship mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 It is a flowchart of a method for mining multi-source heterogeneous data relationships in the present invention;

[0049] Figure 2 A diagram showing the types of multi-source heterogeneous data in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] This embodiment provides a method for mining multi-source heterogeneous data relationships. Figure 1 As shown, the specific steps include:

[0052] S1: Collect multi-source heterogeneous data, including terminal data, transmission link data and network space data, and divide the terminal data, transmission link data and network space data into structured data, semi-structured data and unstructured data, use MongoDB to store structured and semi-structured data, and use MinIO to store unstructured data;

[0053] Specifically, in this embodiment, Figure 2 As shown, terminal data includes server equipment data, smart terminal equipment data and IoT equipment data; transmission link data includes switch data, router data and firewall data; cyberspace data includes platform account identity data, network behavior dynamic data and network service application data. In the process of analyzing and judging attack events, it is necessary to first clarify the scope of the attack, and then comprehensively collect relevant multi-source heterogeneous data. The collected data includes but is not limited to local file systems, relational databases, non-relational databases, devices and application services of Internet application platforms, and third-party API interfaces. Usually, real-time and batch collection is carried out using collection programs or data crawlers.

[0054] Specifically, according to the degree of structuring of the data, MongoDB is used in this embodiment to store structured and semi-structured data, and MinIO is used to store unstructured data, so as to ensure the comprehensiveness and breadth of data acquisition and provide flexible scalability for subsequent data processing and analysis.

[0055] In a specific embodiment, in S2, the process of preprocessing the structured data, semi-structured data and unstructured data to obtain text data includes:

[0056] S21: The process of preprocessing the structured data and semi-structured data is as follows:

[0057] Data deduplication: Identify and remove duplicates by comparing entire strings or calculating hash values ​​of text;

[0058] Missing value filling: fill missing values ​​and missed continuous data by using any of the methods including mean, median, mode, interpolation and model prediction;

[0059] Outlier processing: unify the data units; for example, unify the time format, amount digits, standard address, etc. to unify the collected data units, thereby increasing the comparability of the data.

[0060] Eliminate noisy data: remove HTML tags, special characters and useless spaces to facilitate subsequent natural language processing;

[0061] Dimensionality reduction and normalization: Based on the hash algorithm, static data (such as novels, music, movies, and execution programs) that are not produced by user subjective consciousness are reduced in dimension and normalized, which helps to improve the speed and performance of relationship calculations, ensure that the weights of each feature are equivalent, and facilitate further processing of the subsequent GPT model.

[0062] S22: The process of preprocessing the unstructured data is as follows:

[0063] S221: The unstructured data includes text document data and multimedia data;

[0064] The process of preprocessing the text document data includes:

[0065] Processing Excel files: Use the Apache POI library to read and parse text information in Excel files, that is, read text data in cells and worksheets;

[0066] Word file processing: Use Apache POI library to read and parse text information in Word documents;

[0067] Processing of CSV files: CSV files are simple text files that can be directly read using text file processing methods;

[0068] Processing of PDF files: Use PDFBox to identify the text and layout in PDF files and extract the information; For PDF files containing non-text elements (such as scanned images), use OCR technology to convert the text in the image into editable text format files, and then parse the editable text format files;

[0069] S222: The process of preprocessing the multimedia data includes:

[0070] Image processing: The NeuralTalk tool is used to process images and generate text information. Specifically, the NeuralTalk tool uses technologies such as convolutional neural networks (CNN) and recurrent neural networks (RNN) or long short-term memory networks (LSTM) to encode and decode the visual features of images to achieve conversion from images to text.

[0071] Speech processing: The DeepSpeech2 tool is used to convert speech signals into text information. Specifically, DeepSpeech2 uses end-to-end speech recognition technology, and combines the advantages of CNN and RNN, as well as the application of CTC decoder, to achieve efficient and accurate speech recognition. It uses deep neural networks to extract features, encode and decode audio signals, and finally converts speech signals into text information.

[0072] Video processing: extract the sound channel information from the video, and use the DeepSpeech2 tool to convert the sound channel information into text information; use the frame difference method to extract the picture from the video with a change amplitude exceeding the preset threshold, and perform image semantic analysis to obtain text information. Specifically, when the picture n Frame and m When the frame difference value reaches the preset screen change threshold, get m The NeuralTalk tool is used to extract and recognize natural semantics of the frame. m Frame set to n Frames are detected repeatedly until the video ends. The frame difference method can quickly locate frames with significant changes in the video, improving the efficiency of video processing. The formula of the frame difference method is:

[0073]

[0074] Where: D n ( i , j ) indicates the n Frame and n - m The difference image between frames is at pixel point ( i , j ) value; F n ( i , j ) represents the first n The frame image is at the pixel point ( i , j ) at the pixel value; F n-m ( i , j ) represents the first nm Frame image, that is, nFrame interval m The image of the frame is in pixels ( i , j ) at the pixel value; m It is the number of frames per interval, usually a positive integer greater than or equal to 1, and can be set according to actual needs.

[0075] S3: Performing GPT natural language processing on the preprocessed text data to obtain processed natural language;

[0076] In a specific embodiment, the GPT (Generative Pre-trained Transformer) model is used for natural language processing of the preprocessed text data. The GPT model is based on the Transformer structure, especially its decoder part. In the context of the GPT model, the complete Transformer encoder-decoder architecture is usually not used, but only the decoder is used. Each input sequence is first converted into a sequence of word embeddings, and these embedding vectors represent the semantic information of each word in the sequence; and in the GPT model, the self-attention mechanism works through three main parts: query, key, and value, which are usually obtained by linearly transforming the input embedding vector; based on the concepts of query, key, and value, the model is allowed to dynamically pay attention to the information of other positions according to the information of the current position when processing the sequence. For each position in the sequence, the GPT model calculates the similarity between the query vector of the position and the key vector of other positions (usually calculated by dot product). These similarity scores are called attention scores, which reflect the degree of correlation between different positions. In order to make the attention scores a valid probability distribution, the GPT model performs softmax normalization on them, so that each position will get a normalized attention weight vector, which represents the importance of other positions in the sequence to the current position; finally, the GPT model will perform weighted summation of the value vector according to the attention weight vector to obtain the output representation of each position. This output representation not only contains the information of the current position, but also integrates the relevant information of other positions in the sequence. The self-attention mechanism formula is:

[0077]

[0078] in, Q is the query matrix, K is the key matrix, V is the value matrix, d kis the dimension of the key vector, which is used to scale the dot product result to prevent the gradient from vanishing. The softmax function is used to convert the raw attention scores into a normalized probability distribution, ensuring that the sum of all scores is 1.

[0079] Specifically, by pre-training the GPT model with a large amount of text data, the GPT model learns the intrinsic structure and pattern of the language, so that it can deeply understand the context of the text. When identifying key features, the GPT model will consider the overall context of the text, not just a single word or sentence. This context understanding ability enables the GPT model to accurately capture the key information in the text, thus playing a huge role in mining the relationship between multi-source heterogeneous data of attack entities. For example: (1) The GPT model has a text summary function: The amount of relevant entity data converted into text after pre-processing multi-source heterogeneous data is very large, but the original text, especially the document files and video files in unstructured data, is usually lengthy and contains a lot of non-key information. The GPT model can automatically extract the core points from these texts and generate a concise and clear summary. It not only reduces the amount of data that needs to be processed for data analysis, but also allows the subsequent relationship mining process to more quickly understand the relevance and differences between different data sources, thereby integrating and analyzing more effectively; (2) The GPT model has feature extraction capabilities, including: NER (Named Entity Recognition) is a basic task in natural language processing. The NER system extracts the above entities from unstructured input text and can identify more categories of entity features according to business needs, usually including names, place names, organization names, dates and times, proper nouns, etc. Prompt construction can significantly affect the accuracy, relevance and usefulness of the content generated by the model, and is used to guide LLM to understand the tasks to be performed. For example, "You are an excellent linguist", the specific requirements of the task are "mark all name entities from the given text", etc. The target data type is divided into entity identification, entity attributes, entity relationships, entity behaviors and other data through the training data, which is used for subsequent word segmentation processing.

[0080] S4: Using the HanLP word segmentation tool to perform word segmentation on the processed natural language to obtain entity data phrases; specifically, the entity data phrases include phrase features, phrase parts of speech, and TF-IDF keywords; in this embodiment, after extracting text entities through the GPT model, the word segmentation effect can be optimized;

[0081] Specifically, the phrase position weight includes entity identification weight, entity attribute weight, entity relationship weight, entity behavior weight and other weights;

[0082] The phrase part-of-speech weights include weights of nouns, verbs, adjectives, adverbs, conjunctions, prepositions, pronouns, interjections, and other non-core words.

[0083] S5: determining a phrase position weight, a phrase part-of-speech weight, and a phrase keyword weight based on the entity data phrase, and obtaining a phrase weight based on the phrase position weight, the phrase part-of-speech weight, and the phrase keyword weight;

[0084] In a specific embodiment, in S5, the process of determining the phrase position weight, the phrase part-of-speech weight, and the phrase keyword weight includes:

[0085] S51: Determine phrase position weight;

[0086] The position of a phrase is of great significance to the weight of the phrase and even the similarity of the entire entity. It accurately identifies key identifiers in the text, such as network account ID, email address, phone number, etc. These are usually regarded as strong entity identifiers, which play a decisive role in determining whether two entities are the same. If two entity data are found to have the same strong entity identifier, then the two entities can be considered to be a person or organization. For example: there are two entity-related data sets, which contain information related to the network account ID of mobile terminal A application and information related to the network account ID of Internet platform A application. If the spatial position of the phrase identified by the GPT model belongs to the entity identifier, the two entity Internet data and mobile terminal information can be merged into the same subject. The entity attributes are similar. By comparing the user attribute information such as nicknames and avatars from different Internet platforms (such as A platform and B platform), according to the general habits of users, most of the major platforms use the same avatar, nickname or other similar habits. By fusing multi-source heterogeneous data, it is also possible to further verify whether the two entities point to the same subject. Among them, the entity identifier is an identifier that can directly point to two data sets as the same entity, such as mobile phone number, email address, network account, IP address, browser fingerprint, probe identifier, etc.; based on entity attributes combined with other dimensional data, it can indirectly associate the two data sets as the same entity. Information, entity attributes include nickname, avatar, region, signature, system version, software version, etc.; entity relationship represents the relationship between entity data and other entities to determine whether they are the same entity or similar entities; entity behavior includes the behavioral data generated by the entity converted into natural language through a large model, such as speech, dynamics, habits, likes, information and comments.

[0087] In order to ensure the comparability of weights between different spatial positions and for subsequent processing and user-defined weights, the original spatial position weight values ​​need to be normalized and linearly scaled to [0, 1]. The formula is as follows:

[0088]

[0089] Among them, Position Weight t Indicates that in a given phrase t The normalized position weight, Value( t ) is the weight value of the corresponding spatial position of the phrase in Table 1, p t is with phrase t The corresponding position weight is P t is the set of all position weights related to phrase t. As shown in Table 1, the weights corresponding to different spatial positions given in this embodiment.

[0090] Table 1:

[0091]

[0092] Specifically, users can customize the weight of phrase positions to meet personalized needs based on the diversity of attack types and attack data.

[0093] S52: Determine the word group part-of-speech weight:

[0094] When comparing the similarity of two natural language texts, it is also crucial to include part of speech in the weight calculation of phrases. Part of speech refers to the grammatical role of a word in a sentence, such as noun, verb, adjective, adverb, etc. Using part of speech information can improve the accuracy and efficiency of article similarity comparison, reduce the impact of lexical morphological differences, improve the depth of semantic understanding, filter irrelevant information, enhance feature extraction, and optimize the subsequent weighted Jaccard similarity calculation performance. Among them, the part of speech weight includes the weights of nouns, verbs, adjectives, adverbs, conjunctions, prepositions, pronouns, interjections, and other non-core words. Nouns usually represent specific entities, concepts, or abstract things in articles, and can directly reflect the theme and content of natural language texts. The same or similar nouns appearing in different articles may mean that these articles have similarities in themes or discussion objects. Verbs describe actions, states, or changes. They play the role of connecting subjects and objects and expressing events or behaviors in sentences. The similarity of verbs can be similar in the description of behaviors, events, or processes in natural language texts. Adjectives and adverbs modify nouns and verbs respectively, providing additional information about the modified words. In the natural language text similarity comparison, the similarity of adjectives and adverbs can reflect the similarity of the articles in terms of descriptive details, emotional color or degree. Conjunctions, prepositions and pronouns mainly play the role of connection, introduction or substitution in natural language texts, and are crucial to the overall structure and fluency. Non-core parts of speech such as interjections rarely appear in articles and usually do not directly participate in the expression of the content of the article. It can be used as an auxiliary means of stop word processing.

[0095] In order to ensure that the weights of different parts of speech are comparable and that subsequent processing requires normalization of the part-of-speech weights, the original weight values ​​are linearly scaled to [0, 1], as follows:

[0096]

[0097] Among them, POS Weight s Indicates that in a given phrase s The normalized part-of-speech weight, Value( s ) is the weight value of the word group corresponding to the word group in Table 2, p s is with phrase s The corresponding part-of-speech weight. P s is all with phrase s A set of relevant part-of-speech weights. As shown in Table 2, the weights corresponding to the parts-of-speech of different phrases given in this embodiment.

[0098] Table 2:

[0099]

[0100] S53: Determine the weight of the keyword phrase:

[0101] The calculation formula of the phrase keyword weight is:

[0102]

[0103] Among them, n i,j It is a phrase i In a piece of data related to the attack entity j The number of times it appears in k To traverse a piece of data j The index of all different phrases in ∑ k n k,j Is a piece of data related to the attack entity j The total number of occurrences of all phrases in the data j Total number of phrases;

[0104]

[0105] in, is the total amount of data in the relevant attack entity data, j : t i ∈ d j Contains the phrase t i The number of data; d j Indicates the firstj Item data;

[0106] Multiply TF and IDF to get the phrase keyword weight, i.e. the value of TF-IDF;

[0107]

[0108] In order to eliminate the difference in vector size caused by different document lengths, the weights of phrase keywords are normalized. The normalization formula is:

[0109]

[0110] in, t It is an attack entity related dataset d A phrase in ; TF-IDF( t , d ) is a phrase t In entity related datasets d TF-IDF value in; the denominator is the entity-related dataset d All phrases in t The square root of the sum of the squares of the TF-IDF values ​​of ′, that is, the entity-related dataset d The L2 norm of the TF-IDF vector;

[0111] The phrase weight is obtained based on the phrase position weight, phrase part of speech weight and phrase keyword weight:

[0112] weight = PositionWeight * POSWeight * TF-IDFWeight.

[0113] S6: Calculate entity similarity based on phrase weights and entity data phrases to mine data relationships among multi-source heterogeneous data.

[0114] In a specific embodiment, in S6, the process of calculating entity similarity based on phrase weight and entity data phrase is as follows:

[0115] Each entity data phrase set and its corresponding phrase weight are combined into a weighted vector, expressed as:

[0116]

[0117] in, w n is the weight of each entity data phrase; a n is an entity data phrase; each w n a nRepresents a weighted attribute, namely the weight of a phrase that takes into account the importance of the phrase.

[0118] Since the traditional Jaccard similarity does not consider weights, this embodiment introduces weights and performs similarity calculation based on the improved Jaccard method. The similarity is calculated by using weighted intersection and weighted union. The formula is:

[0119]

[0120] in, w i It is a phrase i The feature weights of a i and b i They are the phrases in different entity data phrase sets. i The presence indicator of .

[0121] Specifically, the improved Jaccard similarity calculation method is combined with phrase weights to compare the similarity between two entity data. By calculating the ratio of the size of the intersection of two sets to the size of the union, the potential association relationship between entities can be more comprehensively explored. In high-dimensional space, when the vector representation of the data contains a large number of zeros (i.e., sparsity), traditional cosine similarity may be greatly affected because they take into account all dimensions (including those with zero values), resulting in unnecessary computational overhead or misleading results. The improved Jaccard similarity calculation method can capture the similarity between two data points in non-zero dimensions while ignoring those zero dimensions, thereby more accurately reflecting the true similarity of the data.

[0122] Specifically, the multi-source heterogeneous data reversal technology MHD-R (Multi-Source Heterogeneous Data Reverse) proposed in this embodiment is used to collect multi-source heterogeneous data such as all terminal devices, transmission links, and network spaces related to the attack event. And these data are correlated and analyzed from multiple dimensional features such as the attacker's attributes, habits, background, relationships, and behaviors, so that multiple scattered attack information such as different spatial locations, different network platforms, and different data structures can be aggregated to form a comprehensive portrait of the attacker entity, and combined with other technical reconnaissance methods to enhance the comprehensive understanding of the attacker. At the same time, the limited association relationships in the entity attributes are ignored, so that the full amount of entity data can participate in the relationship calculation, so that the target entity can be located more comprehensively, accurately, and specifically.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for mining multi-source heterogeneous data relationships, characterized in that: The specific steps include: S1: Collect multi-source heterogeneous data, including terminal data, transmission link data and cyberspace data, and divide the terminal data, transmission link data and cyberspace data into structured data, semi-structured data and unstructured data; S2: Preprocessing the structured data, semi-structured data and unstructured data respectively to obtain text data; S3: performing natural language processing on the preprocessed text data to obtain processed natural language; S4: Perform word segmentation on the processed natural language to obtain entity data phrases; S5: determining a phrase position weight, a phrase part-of-speech weight, and a phrase keyword weight based on the entity data phrase, and obtaining a phrase weight based on the phrase position weight, the phrase part-of-speech weight, and the phrase keyword weight; The phrase position weights include entity identification weights, entity attribute weights, entity relationship weights, entity behavior weights and other weights; S6: Calculate entity similarity based on phrase weights and entity data phrases to mine data relationships from multi-source heterogeneous data; The process of calculating entity similarity based on phrase weight and entity data phrase is: Each entity data phrase set and its corresponding phrase weight are combined into a weighted vector, expressed as: Among them, w n is the weight of each entity data phrase; a n is an entity data phrase; The similarity calculation is based on the improved Jaccard method, and the formula is: Among them, w i is the feature weight of phrase i, a i and b i are the existence indicators of phrase i in different entity data phrase sets respectively.

2. The method for mining multi-source heterogeneous data relationships according to claim 1, characterized in that: The phrase part-of-speech weights include weights of nouns, verbs, adjectives, adverbs, conjunctions, prepositions, pronouns, interjections, and other non-core words.

3. The method for mining multi-source heterogeneous data relationships according to claim 2, characterized in that: The calculation formula of the phrase keyword weight is: Among them, n i,j is the number of times phrase i appears in a piece of data j related to the attack entity; k is the index of all different phrases in a piece of data j; ∑ k n k,j is the total number of occurrences of all phrases in a piece of data j of the relevant attack entity, that is, the total number of phrases in data j; Where |D| is the total amount of data in the relevant attack entity data, j:t i ∈d j Is a phrase containing t i The number of data; d j Indicates the jth piece of data in the relevant attack entity data; Multiply TF and IDF to get the phrase keyword weight, i.e. the value of TF-IDF; <h2 style=";text-align:left;direction:ltr">TF-IDFi, j, TFi, j×IDF<h2 style=";text-align:left;direction:ltr"> i The weight of the phrase keyword is normalized, and the normalization formula is: Where t is a phrase in the attack entity-related dataset d; TF-IDF(t,d) is the TF-IDF value of phrase t in the entity-related dataset d; the denominator is the square root of the sum of the squares of the TF-IDF values ​​of all phrases t′ in the entity-related dataset d, that is, the L2 norm of the TF-IDF vector of the entity-related dataset d; The phrase weight is obtained based on the phrase position weight, phrase part of speech weight and phrase keyword weight: weight=PositionWeight*POSWeight*TF-IDFWeight.

4. The method for mining multi-source heterogeneous data relationships according to claim 3, characterized in that: In S2, the structured data, semi-structured data and unstructured data are preprocessed respectively to obtain text data, including: S21: Preprocessing the structured data and semi-structured data, the process is as follows: Data deduplication: Identify and remove duplicates by comparing entire strings or calculating hash values ​​of text; Missing value filling: fill missing values ​​and missed continuous data by using any of the methods including mean, median, mode, interpolation and model prediction; Outlier processing: unify data units; Eliminate noisy data: remove HTML tags, special characters and useless spaces; Dimensionality reduction and normalization: Dimensionality reduction and normalization of static data based on Hash algorithm; S22: The unstructured data includes text document data and multimedia data; S221: The process of preprocessing the text document data includes: Processing Excel files: Use Apache POI library to read and parse text information in Excel files; Word file processing: Use Apache POI library to read and parse text information in Word documents; Processing of CSV files: directly read through text file processing method; Processing of PDF files: Use PDFBox to identify the text and layout in PDF files and extract the information; For PDF files containing non-text elements, use OCR technology to convert the text in the image into editable text format files, and then parse the editable text format files; S222: The process of preprocessing the multimedia data includes: Image processing: Process the image through the NeuralTalk tool to generate text information; For speech processing: convert speech signals into text information through the DeepSpeech2 tool; Video processing: extract channel information from the video and use DeepSpeech2 to convert the channel information into text information; use the frame difference method to extract the pictures whose picture changes exceed the preset threshold from the video, and perform picture semantic analysis to obtain text information.

5. The method for mining multi-source heterogeneous data relationships according to claim 4, characterized in that: In S3, the GPT model is used to perform natural language analysis on the preprocessed text data.

6. The method for mining multi-source heterogeneous data relationships according to claim 5, characterized in that: In S4, the HanLP word segmentation tool is used to perform word segmentation on the processed natural language.