Attack source identification system and method based on open source network real-time monitoring

Abstract

The invention aims to provide an attack source identification system and method based on open source network real-time monitoring. The system comprises a data acquisition module, a data standardization module, an information addition module, a clustering module and an attack association analysis module. The data acquisition module is used for acquiring IOC data from multiple ways; the data standardization module is used for performing standardization formatting processing on the IOC data; the information adding module is used for adding context information based on the standardized IOC data; the clustering module is used for performing clustering analysis on the IOC data; and the attack association analysis module counts the clustering result and the APT organization association degree and outputs an association result. According to the method, the scattered IOC and the specific APT organization can be associated and attributed, the attack link of the IOC can be possibly output, and deep and actionable context intelligence is provided for threat hunting and event response.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and specifically to an attack source identification system and method based on real-time monitoring of open-source networks. Background Technology

[0002] In the field of cybersecurity, existing technologies for the detection and defense against advanced persistent threats (APTs) mainly focus on the following aspects: Human-led open-source intelligence (OSINT) analysis is the most traditional method. Security analysts manually collect information from public forums, social media, code repositories, security research reports, and other channels, then use their experience to filter, correlate, and attribute information, ultimately extracting attack indicators (IOCs) related to specific APT groups. This process heavily relies on the analyst's personal knowledge, experience, and time investment.

[0003] Monitoring systems based on automated web crawlers and keyword matching utilize web crawlers to scrape content from predefined open-source websites and then match and filter it using predefined keywords (such as malware family names or known APT organization IDs). Their core logic is based on simple string matching, lacking deep semantic understanding and correlation analysis capabilities.

[0004] Rule-based or simple machine learning-based IOC extraction tools can extract structured IOC data (such as IP addresses, domain names, and file hashes) from unstructured text (such as security reports). They typically rely on rules such as regular expressions or use relatively simple named entity recognition models, resulting in limited accuracy and recall, and difficulty in understanding the contextual relationships and tactical intentions between attack metrics.

[0005] Isolated threat intelligence platforms exist in the market, capable of consuming and displaying IOC data. However, the intelligence on these platforms often originates from commercial subscriptions or information-sharing organizations. They lack the ability to autonomously discover, verify, and correlate APT groups from open-source networks in real time, often serving as a data display terminal rather than a source of real-time awareness and production.

[0006] The aforementioned existing technologies, when faced with modern and organized APT attacks, reveal the following core flaws: 1. Severe lack of real-time capability, resulting in delayed early warnings. Defect Description: Manual analysis is inefficient; the time from information emergence to analyst report generation can take hours or even days, making it unable to cope with the rapid evolution of APT attacks. Web crawler-based systems also suffer from significant time delays due to lengthy data processing workflows.

[0007] 2. High information overload and false alarm rate, low degree of automation. Flaw description: Simple keyword matching generates a massive amount of irrelevant information, resulting in a very high false positive rate, requiring extensive manual secondary screening. Analysts are overwhelmed by the sea of ​​information, making it difficult to grasp truly crucial intelligence.

[0008] Relevance to this invention: The "output algorithm" proposed in this invention is not just a simple matching algorithm. It integrates multi-dimensional correlation analysis, behavioral pattern recognition, and confidence assessment, which can effectively filter noise and automatically output high-confidence, verified attack indicators, significantly reducing false alarm rate and dependence on manual intervention.

[0009] 3. Lack of contextual understanding and attribution analysis skills Flaw Description: Most existing tools can only output isolated IOCs (such as an IP address) and cannot automatically explain "which APT group this IP belongs to," "what tactics, techniques, and procedures were used," or "which historical attack campaigns it is associated with." This lack of context significantly diminishes the value of IOCs, making it difficult for security teams to understand the full picture of the threat. Summary of the Invention

[0010] This invention aims to provide an attack source identification system and method based on real-time monitoring of open-source networks. This system and method can associate scattered IOCs with specific APT groups and may output their attack chains, providing in-depth and actionable contextual intelligence for threat hunting and incident response.

[0011] The technical solution of the present invention is as follows: The attack source identification system based on real-time monitoring of open-source networks includes a data acquisition module, a data standardization module, an information appending module, a clustering module, and an attack correlation analysis module. The data acquisition module is used to monitor preset target websites using web crawlers, crawl the content of posts on the websites, and extract possible cybersecurity threat indicator (IOC) data from the text using natural language processing (NLP); or obtain IOC data from external threat intelligence platforms through API interfaces; or extract cybersecurity threat indicator (IOC) data from virus samples input by the module itself; the cybersecurity threat indicator (IOC) data includes the original string, source, and collection time; these data are preprocessed to remove duplicate and incomplete data to obtain initial IOC data; The data standardization module is used to standardize and format the initial IOC data to obtain standardized IOC data. The information appending module is used to add contextual information based on standardized IOC data to obtain enriched IOC data. The clustering module is used to perform cluster analysis on the enriched IOC data, identify the enriched IOC data from the same source, group them into one category, traverse all the enriched IOC data to obtain each IOC data cluster; and count the IOC data that appears repeatedly in each IOC data cluster. The more times it appears, the higher the weight of that IOC data is set. The attack association analysis module calculates the similarity score between each IOC data cluster and each APT organization stored in the database, sets a threshold, associates clusters with similarity scores exceeding the threshold with APT organizations, calculates the confidence level for each association result, and records the association evidence; then it outputs the association results, including the association relationship between the cluster and the APT organization, the confidence level, and the evidence.

[0012] The data acquisition module uses a streaming processing rack for data acquisition and preprocessing, and its cumulative data acquisition volume formula is as follows: (1) in, The arrival rate of data at time τ is represented by D(t), which is the cumulative amount of data collected within the time interval [0, t].

[0013] The data standardization module described above performs the following steps to standardize and format the initial IOC data: Use regular expressions to match the types of IP addresses, domain names, hashes, and URLs; standardize and format the IOC data, unifying IP addresses to dotted decimal, domain names to lowercase, and hash values ​​to lowercase; calculate the MD5 hash value for each IOC data as a unique standard.

[0014] The standardized IOC data includes type, value, unique identifier, and original data source.

[0015] The context information added in the information appending module includes: for IP addresses, querying geographical location information through the GeoIP database; querying internal or external reputation systems to obtain the IOC's reputation score; for domain names and IP names, querying WHOIS information to obtain registrant and registration time information; and for file hashes, querying file type, size, and whether it exists in the malware database.

[0016] The clustering method used in the clustering module is the DBSCAN algorithm, and the algorithm community discovery algorithm is used for the statistical analysis of recurring IOC data. The IOC data clusters obtained by clustering in the clustering module are expressed by the following multi-dimensional feature vector space formula: (2) Where v is a feature vector of an attacking entity. n is the number of feature dimensions. Let i be an n-dimensional real vector space, where each dimension i corresponds to an attack tactic or technical attribute. .

[0017] The similarity calculation process in the attack association analysis module is as follows: The feature vector of the attacking entity. Let be the feature vector template of the j-th APT organization, then the similarity between them is... It can be represented as: (3) in, It is a cosine similarity metric function. It is used to calculate the feature vector of the attacking entity. With the feature vector templates of each APT organization Similarity between This enables attribution judgment of attack indicators. When When the similarity threshold is ≥θ, the attack metrics are attributed to the APT group j. If <θ, then it is determined to be of unknown origin, i.e. (4).

[0018] In the attack correlation analysis module, the confidence calculation formula is as follows: (5) Where x is the input feature vector, including data source credibility, number of cross-validation iterations, etc. It is the weight vector learned through the training process, σ(*) is the activation function, and C is the confidence score.

[0019] The attack source identification system based on open-source network real-time monitoring also includes a visualization interface display module, which is used to display the output results of the attack correlation analysis module in real time.

[0020] This invention also discloses an attack source identification method based on real-time monitoring of open-source networks, which applies the aforementioned attack source identification system based on real-time monitoring of open-source networks and includes the following steps: A. The data acquisition module collects data from multiple sources: a. Monitor the preset target websites using web crawlers, crawl the content of posts on the websites, and extract possible cybersecurity threat indicators (IOCs) from the text using natural language processing (NLP). b. Obtain IOC data from external threat intelligence platforms via API interfaces; c. Extract cybersecurity threat indicator (IOC) data from the virus sample you input; the cybersecurity threat indicator (IOC) data includes the original string, source, and collection time; d. Preprocess the collected data to obtain initial IOC data by removing duplicate and incomplete data, and then send it to the data standardization module; B. The data standardization module performs standardization and formatting processing on the initial IOC data to obtain standardized IOC data, which is then input into the information addition module. C. The information addition module is used to add contextual information based on standardized IOC data to obtain enriched IOC data, which is then input into the clustering module. D. The clustering module is used to perform cluster analysis on the enriched IOC data, identify enriched IOC data from the same source, group them into one category, traverse all riched IOC data, and obtain each IOC data cluster; for IOC data data that appear repeatedly in each IOC data cluster, the more times they appear, the higher the weight of that IOC data is assigned; the IOC data clusters and the weights of the calculated IOC data are input into the attack correlation analysis module; E. The attack association analysis module calculates the similarity score between each IOC data cluster and each APT organization stored in the database, sets a threshold, associates clusters with similarity scores exceeding the threshold with APT organizations, calculates the confidence level for each association result, and records the association evidence; then outputs the association results, including the association relationship between the cluster and the APT organization, the confidence level, and the evidence.

[0021] Technical advantages of the present invention 1. The advantages of open-source real-time network monitoring Traditional methods rely on scheduled crawlers or manual browsing, resulting in long data update cycles and difficulty handling complex network environments with dynamic loading and anti-crawler measures. The advancement of this invention lies in its streaming data processing architecture, employing stream processing technology to continuously monitor and ingest open-source information sources, achieving data acquisition and transmission at the second or even millisecond level. Using a multi-source heterogeneous data fusion engine, it can simultaneously process open-source information in various formats, including text, code, and network metadata, and perform unified structured processing, solving the core challenge of complex data sources and inconsistent formats. Intelligent crawling and adaptive monitoring are also key features. The crawler possesses self-learning capabilities, dynamically adjusting monitoring sites and keywords based on monitoring targets, ensuring the breadth and sustainability of intelligence gathering.

[0022] 2. The advanced nature of APT group attack indicator output methods Traditional methods rely on regular expressions to extract Indices of Computation (IOCs) or simple keyword lists for matching, which fail to understand semantics, resulting in high false positive rates and an inability to correlate attribution. The advancement of this invention lies in its application of natural language processing and deep semantic understanding. The method can understand cybersecurity terminology and jargon (e.g., distinguishing between "Cobalt Strike" as malware and "Cobalt Strike" as a legitimate tool). It can extract entities and relationships from unstructured text to construct a knowledge graph. Utilizing a dynamic behavior pattern recognition and association engine, the algorithm has a built-in or learned knowledge base of attack tactics, techniques, and procedures (TTPs) from known APT organizations. When new attack indicators are extracted from open-source information, the algorithm performs multi-dimensional correlation matching with the TTPs in the knowledge base to provide attribution judgments.

[0023] 3. The systematic and advanced nature of the overall architecture Self-evolution and adaptive capabilities: As an organic whole, the entire system continuously optimizes its monitoring strategies, semantic understanding models, and attribution algorithms by constantly receiving new data and feedback. This means that the system possesses greater vitality and adaptability when facing the ever-changing tactics of APT organizations.

[0024] The beneficial effects of this invention are as follows: This invention, by monitoring open-source networks in real time, can detect clues in the early stages of the attack chain (such as the attacker's weapon preparation and reconnaissance phases), thus buying valuable golden response time for the defender.

[0025] This invention automatically outputs high-value, high-confidence attack indicators through algorithms, greatly improving analysis efficiency and enabling professionals to focus on the highest-risk threats, effectively solving the information overload problem faced by security analysts.

[0026] This invention outputs not only isolated IP addresses or hash values, but also intelligence accompanied by APT group tags and tactical context. This makes defensive measures more targeted; for example, identifying the specific APT group allows for prediction of its next possible action.

[0027] This invention can free security analysts from repetitive and tedious information gathering and initial screening work, allowing them to engage in more advanced strategic analysis, threat hunting, and strategy formulation, achieving a perfect combination of human and machine advantages and showing promising application prospects. Detailed Implementation

[0028] The present invention will now be described in detail. Example

[0029] An attack source identification system based on open-source network real-time monitoring includes a data acquisition module, a data standardization module, an information appending module, a clustering module, and an attack correlation analysis module. The data acquisition module is used to monitor preset target websites using web crawlers, crawl the content of posts on the websites, and extract possible cybersecurity threat indicator (IOC) data from the text using natural language processing (NLP); or obtain IOC data from external threat intelligence platforms through API interfaces; or extract cybersecurity threat indicator (IOC) data from virus samples input by the module itself; the cybersecurity threat indicator (IOC) data includes the original string, source, and collection time; these data are preprocessed to remove duplicate and incomplete data to obtain initial IOC data; The formula for the cumulative amount of data collected for data acquisition and preprocessing is: (1) in, The arrival rate of data at time τ is represented by D(t), which is the cumulative amount of data collected within the time interval [0, t].

[0030] The data standardization module is used to standardize and format the initial IOC data to obtain standardized IOC data; the standardized IOC data includes type, value, unique identifier, and original data source.

[0031] Standardizing and formatting the initial IOC data includes the following steps: Use regular expressions to match the types of IP addresses, domain names, hashes, and URLs; standardize and format the IOC data, unifying IP addresses to dotted decimal, domain names to lowercase, and hash values ​​to lowercase; calculate the MD5 hash value for each IOC data as a unique standard.

[0032] The aforementioned information appending module is used to add contextual information based on standardized IOC data to obtain enriched IOC data. The contextual information includes: for IP addresses, querying geographical location information through the GeoIP database; querying internal or external reputation systems to obtain the IOC's reputation score; for domain names and IP names, querying WHOIS information to obtain registrant and registration time information; and for file hashes, querying file type, size, and whether it exists in the malware database.

[0033] The clustering module is used to perform cluster analysis on the enriched IOC data, identify the enriched IOC data from the same source, group them into one category, traverse all the enriched IOC data to obtain each IOC data cluster; and count the IOC data that appears repeatedly in each IOC data cluster. The more times it appears, the higher the weight of that IOC data is set. The clustering method used is the DBSCAN algorithm, and the community detection algorithm is used for the statistical analysis of recurring IOC data. The IOC data clusters obtained by clustering in the clustering module are expressed by the following multi-dimensional feature vector space formula: (2) Where v is a feature vector of an attacking entity. , where n is the number of feature dimensions. Let i be an n-dimensional real vector space, where each dimension i corresponds to an attack tactic or technical attribute. .

[0034] The attack association analysis module calculates the similarity score between each IOC data cluster and each APT organization stored in the database, sets a threshold, associates clusters with similarity scores exceeding the threshold with APT organizations, calculates the confidence level for each association result, and records the association evidence; then it outputs the association results, including the association relationship between the cluster and the APT organization, the confidence level, and the evidence.

[0035] The similarity calculation process is as follows: The feature vector of the attacking entity. Let be the feature vector template of the j-th APT organization, then the similarity between them is... It can be represented as: (3) in, It is a cosine similarity metric function. It is used to calculate the feature vector of the attacking entity. With the feature vector templates of each APT organization similarity between This enables attribution judgment of attack indicators. When When the similarity threshold is ≥θ, the attack metrics are attributed to the APT group j. If <θ, then it is determined to be of unknown origin, i.e. (4).

[0036] In the attack correlation analysis module, the confidence calculation formula is as follows: (5) Where x is the input feature vector, including data source credibility, number of cross-validation iterations, etc. It is the weight vector learned through the training process, σ(*) is the activation function, and C is the confidence score.

[0037] The attack source identification system based on open-source network real-time monitoring also includes a visualization interface display module, which is used to display the output results of the attack correlation analysis module in real time.

[0038] The attack source identification method based on open-source real-time network monitoring includes the following steps: A. The data acquisition module collects data from multiple sources: a. Monitor the preset target websites using web crawlers, crawl the content of posts on the websites, and extract possible cybersecurity threat indicators (IOCs) from the text using natural language processing (NLP). b. Obtain IOC data from external threat intelligence platforms via API interfaces; c. Extract cybersecurity threat indicator (IOC) data from the virus sample you input; the cybersecurity threat indicator (IOC) data includes the original string, source, and collection time; d. Preprocess the collected data to obtain initial IOC data by removing duplicate and incomplete data, and then send it to the data standardization module; B. The data standardization module performs standardization and formatting processing on the initial IOC data to obtain standardized IOC data, which is then input into the information addition module. C. The information addition module is used to add contextual information based on standardized IOC data to obtain enriched IOC data, which is then input into the clustering module. D. The clustering module is used to perform cluster analysis on the enriched IOC data, identify enriched IOC data from the same source, group them into one category, traverse all riched IOC data, and obtain each IOC data cluster; for IOC data data that appear repeatedly in each IOC data cluster, the more times they appear, the higher the weight of that IOC data is assigned; the IOC data clusters and the weights of the calculated IOC data are input into the attack correlation analysis module; E. The attack association analysis module calculates the similarity score between each IOC data cluster and each APT organization stored in the database, sets a threshold, associates clusters with similarity scores exceeding the threshold with APT organizations, calculates the confidence level for each association result, and records the association evidence; then outputs the association results, including the association relationship between the cluster and the APT organization, the confidence level, and the evidence.

[0039] Those skilled in the art should recognize that the above embodiments are merely illustrative of the present invention and are not intended to limit the present invention. Any variations or modifications to the above embodiments that are within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. An attack source identification system based on open-source network real-time monitoring, comprising a data acquisition module, a data standardization module, an information appending module, a clustering module, and an attack correlation analysis module, characterized in that: The data acquisition module is used to monitor preset target websites using web crawlers, crawl the content of posts on the websites, and extract possible cybersecurity threat indicator (IOC) data from the text using natural language processing (NLP); or obtain IOC data from external threat intelligence platforms through API interfaces; or extract cybersecurity threat indicator (IOC) data from virus samples input by the module itself; the cybersecurity threat indicator (IOC) data includes the original string, source, and collection time; these data are preprocessed to remove duplicate and incomplete data to obtain initial IOC data; The data standardization module is used to standardize and format the initial IOC data to obtain standardized IOC data. The information appending module is used to add contextual information based on standardized IOC data to obtain enriched IOC data. The clustering module is used to perform cluster analysis on the enriched IOC data, identify the enriched IOC data from the same source, group them into one category, traverse all the enriched IOC data to obtain each IOC data cluster; and count the IOC data that appears repeatedly in each IOC data cluster. The more times it appears, the higher the weight of that IOC data is set. The attack association analysis module calculates the similarity score between each IOC data cluster and each APT organization stored in the database, sets a threshold, associates clusters with similarity scores exceeding the threshold with APT organizations, calculates the confidence level for each association result, and records the association evidence; then it outputs the association results, including the association relationship between the cluster and the APT organization, the confidence level, and the evidence.

2. The attack source identification system based on real-time open-source network monitoring as described in claim 1, characterized in that: The data acquisition module uses a streaming processing rack for data acquisition and preprocessing, and its cumulative data acquisition volume formula is as follows: （1） in, The arrival rate of data at time τ is represented by D(t), which is the cumulative amount of data collected within the time interval [0, t].

3. The attack source identification system based on real-time open-source network monitoring as described in claim 1, characterized in that: The data standardization module described above performs the following steps to standardize and format the initial IOC data: Use regular expressions to match the types of IP addresses, domain names, hashes, and URLs; standardize and format the IOC data, unifying IP addresses to dotted decimal, domain names to lowercase, and hash values ​​to lowercase; calculate the MD5 hash value for each IOC data as a unique standard.

4. The attack source identification system based on real-time open-source network monitoring as described in claim 1, characterized in that: The standardized IOC data includes type, value, unique identifier, and original data source.

5. The attack source identification system based on real-time open-source network monitoring as described in claim 1, characterized in that: The context information added in the information appending module includes: for IP addresses, querying geographical location information through the GeoIP database; querying internal or external reputation systems to obtain the IOC's reputation score; for domain names and IP names, querying WHOIS information to obtain registrant and registration time information; and for file hashes, querying file type, size, and whether it exists in the malware database.

6. The attack source identification system based on real-time open-source network monitoring as described in claim 1, characterized in that: The clustering method used in the clustering module is the DBSCAN algorithm, and the algorithm community discovery algorithm is used for the statistical analysis of recurring IOC data. The IOC data clusters obtained by clustering in the clustering module are expressed by the following multi-dimensional feature vector space formula: （2） Where v is a feature vector of an attacking entity. n is the number of feature dimensions. Let i be an n-dimensional real vector space, where each dimension i corresponds to an attack tactic or technical attribute. .

7. The attack source identification system based on real-time open-source network monitoring as described in claim 1, characterized in that: The similarity calculation process in the attack association analysis module is as follows: The feature vector of the attacking entity. Let be the feature vector template of the j-th APT organization, then the similarity between them is... It can be represented as: （3） in, It is a cosine similarity metric function. It is used to calculate the feature vector of the attacking entity. With the feature vector templates of each APT organization Similarity between This enables attribution judgment of attack indicators. When When the similarity threshold is ≥θ, the attack metrics are attributed to the APT group j. If <θ, then it is determined to be of unknown origin, i.e. （4）。 8. The attack source identification system based on real-time open-source network monitoring as described in claim 1, characterized in that: In the attack correlation analysis module, the confidence calculation formula is as follows: （5） Where x is the input feature vector, including data source credibility, number of cross-validation iterations, etc. It is the weight vector learned through the training process, σ(*) is the activation function, and C is the confidence score.

9. The attack source identification system based on real-time open-source network monitoring as described in claim 1, characterized in that: It also includes a visualization interface display module, which is used to display the output results of the attack correlation analysis module in real time.

10. A method for identifying attack sources based on real-time monitoring of open-source networks, employing the attack source identification system based on real-time monitoring of open-source networks as described in any one of claims 1-9, characterized in that, Includes the following steps: A. The data acquisition module collects data from multiple sources: a. Monitor the preset target websites using web crawlers, crawl the content of posts on the websites, and extract possible cybersecurity threat indicators (IOCs) from the text using natural language processing (NLP). b. Obtain IOC data from external threat intelligence platforms via API interfaces; c. Extract cybersecurity threat indicator (IOC) data from the virus sample you input; the cybersecurity threat indicator (IOC) data includes the original string, source, and collection time; d. Preprocess the collected data to obtain initial IOC data by removing duplicate and incomplete data, and then send it to the data standardization module; B. The data standardization module performs standardization and formatting processing on the initial IOC data to obtain standardized IOC data, which is then input into the information addition module. C. The information addition module is used to add contextual information based on standardized IOC data to obtain enriched IOC data, which is then input into the clustering module. D. The clustering module is used to perform cluster analysis on the enriched IOC data, identify enriched IOC data from the same source, group them into one category, traverse all riched IOC data, and obtain each IOC data cluster; for IOC data data that appear repeatedly in each IOC data cluster, the more times they appear, the higher the weight of that IOC data is assigned; the IOC data clusters and the weights of the calculated IOC data are input into the attack correlation analysis module; E. The attack association analysis module calculates the similarity score between each IOC data cluster and each APT organization stored in the database, sets a threshold, associates clusters with similarity scores exceeding the threshold with APT organizations, calculates the confidence level for each association result, and records the association evidence; then outputs the association results, including the association relationship between the cluster and the APT organization, the confidence level, and the evidence.

Images