Comprehensive threat intelligence platform based on deep learning and big data fusion

Through a comprehensive threat intelligence platform integrating deep learning and big data, traditional threat detection is solved, and efficient threat intelligence collection, analysis and sharing is achieved, and network security protection capabilities are improved.

CN119026116AInactive Publication Date: 2024-11-26SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411002557.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional threat detection and response mechanisms are difficult to effectively respond to new security challenges such as advanced persistent threats, zero-day attacks and ransomware. The threat intelligence system faces problems such as single intelligence sources, inefficient analysis, and lagging response.

Method used

Adopt a comprehensive threat intelligence platform based on the integration of deep learning and big data, including multi-dimensional threat intelligence acquisition and fusion module, deep learning-driven threat identification and analysis engine, and dynamic threat warning and emergency response system, through automatic collection, data cleaning, and features Technologies such as extraction, abnormal detection and threat association analysis can achieve efficient collection, analysis and sharing of threat intelligence.

Benefits of technology

It improves the efficiency of acquisition, analysis accuracy and application value of threat intelligence, can effectively warn and resist various cyber attacks, support dynamic adaptation to changes in the security environment, and provide organizations with continuous upgraded network security guarantees.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026116A_ABST
    Figure CN119026116A_ABST
Patent Text Reader

Abstract

The invention provides a comprehensive threat intelligence platform based on deep learning and big data fusion, which belongs to the technical field of information security and comprises a multi-dimensional threat intelligence acquisition and fusion module, a deep learning driven threat identification and analysis engine, a dynamic threat early warning and emergency response system and a self-learning and optimization mechanism. Through highly automatic and intelligent means, real-time monitoring, accurate analysis, early warning response and strategy optimization of security threats in a whole network range are realized, and all-around and multi-level network security protection support is provided for various organizations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a comprehensive threat intelligence platform based on the fusion of deep learning and big data. Background Art

[0002] With the acceleration of digital transformation, the cybersecurity situation is becoming increasingly complex and severe. Traditional threat detection and response mechanisms are limited by technical bottlenecks and often cannot effectively respond to new security challenges such as advanced persistent threats (APTs), zero-day attacks, and ransomware. As a key element in improving security defense capabilities, threat intelligence is valuable in that it can reveal potential threats in a timely manner and guide the formulation of defense strategies. However, the current threat intelligence system generally faces problems such as a single intelligence source, low analysis efficiency, and delayed response. Technological innovation is urgently needed to meet the growing security protection needs. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a comprehensive threat intelligence platform based on the fusion of deep learning and big data to solve the deficiencies in the existing technology, improve the efficiency of threat intelligence collection, analysis, integration and sharing, and thus enhance network security protection capabilities.

[0004] The technical solution of the present invention is:

[0005] A comprehensive threat intelligence platform based on the fusion of deep learning and big data, including:

[0006] Multi-dimensional threat intelligence collection and fusion module, which automatically collects security information, combines, correlates and combines data to form a threat intelligence database;

[0007] A deep learning-driven threat identification and analysis engine extracts features and uses them for classification or regression tasks; hidden states are updated to enable information transfer and preservation;

[0008] Dynamic threat warning and emergency response system, real-time warning, emergency response suggestion generation and feedback loop optimization;

[0009] Self-learning and optimization module, online learning model self-adjustment and user feedback loop to continuously improve system performance.

[0010] Furthermore,

[0011] The multi-dimensional threat intelligence collection and fusion module includes at least two different types of data collection sub-modules and data fusion algorithms.

[0012] Deep learning-driven threat identification and analysis engine, including feature extraction, anomaly detection, classification model, and threat correlation analysis sub-modules.

[0013] Going further,

[0014] Multi-dimensional threat intelligence collection and fusion module, including

[0015] Diversification of data sources: Design and implement a distributed crawler network, divide a crawler task into several subtasks, and distribute them to multiple computer nodes for parallel processing; automatically collect security-related information, and at the same time, combine data leakage and public data mining methods to discover potential security risks and risk points;

[0016] Data cleaning and fusion: Use big data cleaning technology to process and prepare the collected large-scale data to remove noise, handle missing values, and solve data inconsistencies. Use data fusion algorithms to combine, correlate, and combine data from multiple sensors or information sources to obtain information and form a unified threat intelligence database.

[0017] Deep learning-driven threat identification and analysis engine, including:

[0018] Feature extraction and representation learning: The autoencoder in deep learning is used to map the input data to a low-dimensional hidden layer, while the decoder maps the representation of the hidden layer back to the original input space. The convolutional neural network (CNN) extracts useful features by convolving the input data with a set of learnable parameters, and uses these features for classification or regression tasks.

[0019] Anomaly detection and classification model: Based on the recurrent neural network (RNN), a memory mechanism is introduced to enable the neural network to process sequence data and keep the memory of past information;

[0020] Threat association analysis: Through the graph neural network GNN, neural networks are used to learn graph structure data, extract and discover features and patterns in graph structure data to meet the needs of graph learning tasks such as clustering, classification, prediction, segmentation, and generation. Learn the representation vectors of nodes and edges so that various tasks can be performed on the graph to build association maps between threat entities, analyze attack links and malicious group activities, and reveal hidden threat relationships.

[0021] in,

[0022] The workflow of feature extraction and representation learning uses convolution, pooling and full connection technologies to extract features from massive threat intelligence, automatically extract features or attributes from raw data, and conduct model training, classification and clustering to achieve high-dimensional vector representation of malicious code, network behavior and vulnerability features, thereby enhancing recognition accuracy.

[0023] Anomaly detection and classification models, including

[0024] The hidden state of each time step is updated according to the input and the hidden state of the previous time step, thereby realizing the transmission and preservation of information, allowing the recurrent neural network RNN ​​to capture the contextual relationship and model the time series data. The long short-term memory network LSTM controls the flow of information by introducing three gated units, namely the input gate, the forget gate and the output gate, and maintains a cell state internally to memorize long-term dependencies to deal with long sequence dependency problems, prevent gradient disappearance problems, and have good learning ability to build a time series analysis model, monitor network traffic and log records in real time, and identify abnormal behavior patterns; at the same time, use deep learning classifiers to accurately classify threat types.

[0025] The dynamic threat warning and emergency response system includes real-time warning: establishing a warning model based on machine learning to predict and identify potential risks or problems, analyzing and modeling through mathematical, statistical and machine learning methods to discover potential risks or problems and issue warnings in advance; immediately assessing the urgency and potential impact of identified threats, and providing real-time notifications through API, email, and SMS.

[0026] Emergency response recommendations: Combined with historical case libraries and knowledge bases, it contains expertise, experience, best practices, case studies, and solutions in the security field or industry, so that this knowledge can be accessed and utilized to support decision-making, solve problems, or improve work efficiency, generate targeted emergency response recommendations and mitigation measures, and assist personnel to take quick action.

[0027] The beneficial effects of the present invention are

[0028] The comprehensive threat intelligence platform of the present invention, through the deep integration of deep learning and big data, realizes deep mining, rapid response and intelligent protection of network security threats, greatly improving the acquisition efficiency, analysis accuracy and application value of threat intelligence. The platform can not only effectively warn and resist various known and unknown network attacks, but also support dynamic adaptation to changes in the security environment, providing organizations with continuously upgraded network security protection. In addition, the modular design of the platform facilitates flexible expansion and customization according to actual needs, meeting the specific security needs of users of different industries and scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the workflow of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] This paper proposes an innovative comprehensive threat intelligence platform that integrates the efficient processing capabilities of deep learning algorithms and the extensive data integration capabilities of big data technologies, aiming to build a closed-loop system from data collection, processing, analysis to application.

[0032] To realize the technical content, the working steps of the present invention are as follows:

[0033] 1. Multi-dimensional threat intelligence collection and fusion module

[0034] 1) Diversification of data sources: Design and implement a distributed crawler network that divides a crawler task into multiple subtasks and distributes them to multiple computer nodes for parallel processing. This technology can greatly improve the efficiency and speed of crawlers while reducing the load and risk of a single node. Automatically collect security-related information from multiple dimensions such as open networks, private data sharing platforms, social media, deep web / dark web, etc., and consider using information security collection and monitoring technologies such as network reconnaissance, social engineering, and permission testing to obtain more comprehensive security information. At the same time, combined with methods such as data leakage and public data mining, potential security risks and risk points can be discovered to ensure the comprehensiveness of intelligence sources.

[0035] 2) Data cleaning and fusion: Use big data cleaning technology to process and prepare the collected large-scale data to remove noise, handle missing values, solve data inconsistencies and other problems, so as to ensure the quality and availability of data. Through data fusion algorithms such as pixel-level fusion, feature-level fusion, and decision-level fusion, data from multiple sensors or information sources are combined, correlated, and combined to obtain more accurate and comprehensive information to form a unified threat intelligence database.

[0036] 2. Deep learning-driven threat identification and analysis engine

[0037] 1) Feature extraction and representation learning: The autoencoder in deep learning is used to map the input data to a low-dimensional hidden layer, while the decoder maps the representation of the hidden layer back to the original input space. The convolutional neural network (CNN) extracts useful features by convolving the input data with a set of learnable parameters, and uses these features for classification or regression tasks. Its workflow can be divided into three steps: convolution, pooling, and full connection technologies are used to extract features from massive threat intelligence. Meaningful features or attributes are automatically extracted from the raw data to better perform model training, classification, clustering and other tasks. High-dimensional vector representation of malicious code, network behavior, and vulnerability features is achieved to enhance recognition accuracy.

[0038] 2) Anomaly detection and classification model: Based on the recurrent neural network (RNN), a "memory" mechanism is introduced to enable the neural network to process sequence data and maintain memory of past information. The hidden state of each time step is updated according to the input and the hidden state of the previous time step, thereby realizing the transmission and preservation of information. This allows RNN to capture contextual relationships and model time series data. The long short-term memory network (LSTM) controls the flow of information by introducing three gated units (input gate, forget gate, and output gate), and maintains a cell state internally to memorize long-term dependencies. It can effectively handle long sequence dependency problems, prevent gradient vanishing problems, and have good learning capabilities to build time series analysis models, monitor network traffic, log records, etc. in real time, and identify abnormal behavior patterns; at the same time, use deep learning classifiers to accurately classify threat types.

[0039] 3) Threat association analysis: Through graph neural networks (GNNs), neural networks are used to learn graph structured data, extract and discover features and patterns in graph structured data to meet the requirements of graph learning tasks such as clustering, classification, prediction, segmentation, and generation. The representation vectors of nodes and edges are learned so that various tasks can be performed on the graph to build association maps between threat entities, analyze attack links and malicious group activities, reveal hidden threat relationships, and provide deep insights for security policy formulation.

[0040] 3. Dynamic threat warning and emergency response system

[0041] 1) Real-time warning: Establish early warning models based on machine learning to predict and identify potential risks or problems. These models are usually based on a large amount of historical data and current data, and are analyzed and modeled through mathematical, statistical and machine learning methods to discover potential risks or problems and issue warnings in advance. Immediately assess the urgency and potential impact of identified threats, and notify the security team in real time through multiple channels such as API, email, and SMS.

[0042] 2) Emergency response recommendations: Combined with the historical case library and expert knowledge base, it contains professional knowledge, experience, best practices, case studies, solutions, etc. in the security field or industry, so that members within the organization can easily access and use this knowledge to support decision-making, solve problems or improve work efficiency. Generate targeted emergency response recommendations and mitigation measures to assist security personnel in taking quick action.

[0043] 4. Self-learning and optimization mechanism

[0044] 1) Model self-evolution: Introducing an online learning mechanism to continuously train and update the model as data flows gradually arrive, so that it can adapt to data changes and new situations in a timely manner. This enables the system to automatically adjust model parameters based on newly emerging threat samples, and continuously optimize recognition accuracy and response speed.

[0045] 2) User feedback loop: Design a user feedback interface to collect the security team's evaluation of warnings and analysis results. We need to clarify the warning type and goal, design a feedback mechanism, determine the feedback content, encourage active participation, and regularly collect and analyze feedback to ensure that all team members are clear about the type of warning, purpose, and expected response, which are then used to iteratively optimize model performance and improve user experience.

[0046] 3) Customized security information: Based on the needs of the enterprise and the specific industry background, the platform can provide customized information and suggestions for the enterprise. For example, for enterprises in the financial industry, the platform can provide information and suggestions related to financial fraud; for enterprises in the manufacturing industry, the platform can provide information and suggestions related to intellectual property protection. This customized information helps enterprises better understand the threat environment they are in and improve their ability to respond to threats.

[0047] 4) Integration and expansion: The platform can be integrated with various security tools and systems to provide a more comprehensive network security solution. For example, the platform can be integrated with intrusion detection systems and security incident response systems to achieve real-time threat detection and response. In addition, the platform can be expanded and customized according to user needs to meet the specific needs of different industries and enterprises.

[0048] 5) Threat intelligence sharing and cooperation: On the platform, different companies can share threat intelligence and cooperate with other companies. This cooperation helps companies to jointly deal with network threats and improve the network security level of the entire industry.

[0049] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. Comprehensive threat intelligence platform based on the fusion of deep learning and big data, It is characterized in that include: Multi-dimensional threat intelligence collection and fusion module, which automatically collects security information, combines, correlates and combines data to form a threat intelligence database; A deep learning-driven threat identification and analysis engine extracts features and uses them for classification or regression tasks; hidden states are updated to enable information transfer and preservation; Dynamic threat warning and emergency response system, real-time warning, emergency response suggestion generation and feedback loop optimization; Self-learning and optimization module, online learning model self-adjustment and user feedback loop to continuously improve system performance.

2. The platform according to claim 1, It is characterized in that The multi-dimensional threat intelligence collection and fusion module includes at least two different types of data collection sub-modules and data fusion algorithms; Deep learning-driven threat identification and analysis engine, including feature extraction, anomaly detection, classification model, and threat correlation analysis sub-modules.

3. The platform according to claim 2, It is characterized in that Multi-dimensional threat intelligence collection and fusion module, including Diversification of data sources: Design and implement a distributed crawler network, divide a crawler task into several subtasks, and assign them to several computer nodes for parallel processing; automatically collect security-related information, and at the same time, combine data leakage and public data mining methods to discover potential security risks and risk points; Data cleaning and fusion: Use big data cleaning technology to process and prepare the collected large-scale data to remove noise, handle missing values, and solve data inconsistency problems; use data fusion algorithms to combine, correlate, and combine data from sensors or information sources to obtain information and form a unified threat intelligence database.

4. The platform according to claim 2, It is characterized in that Deep learning-driven threat identification and analysis engine, including: Feature extraction and representation learning: The autoencoder in deep learning is used to map the input data to a low-dimensional hidden layer, while the decoder maps the representation of the hidden layer back to the original input space. The convolutional neural network (CNN) extracts useful features by convolving the input data with a set of learnable parameters, and uses these features for classification or regression tasks. Anomaly detection and classification model: Based on the recurrent neural network (RNN), a memory mechanism is introduced to enable the neural network to process sequence data and keep the memory of past information; Threat association analysis: Through the graph neural network GNN, neural networks are used to learn graph structure data, extract and discover features and patterns in graph structure data, and learn the representation vectors of nodes and edges that meet the requirements of graph learning tasks, so as to perform various tasks on the graph to build association maps between threat entities, analyze attack links and malicious group activities, and reveal hidden threat relationships.

5. The platform according to claim 4, It is characterized in that The workflow of feature extraction and representation learning uses convolution, pooling and full connection technologies to extract features from massive threat intelligence, automatically extract features or attributes from raw data, and conduct model training, classification and clustering to achieve high-dimensional vector representation of malicious code, network behavior and vulnerability features, thereby enhancing recognition accuracy.

6. The platform according to claim 4, It is characterized in that The anomaly detection and classification model specifically includes that the hidden state of each time step will be updated according to the input and the hidden state of the previous time step, so as to realize the transmission and preservation of information, so that the recurrent neural network RNN ​​can capture the contextual relationship and model the time series data. The long short-term memory network LSTM controls the flow of information by introducing three gate units, namely the input gate, the forget gate and the output gate, and maintains a cell state internally to memorize long-term dependencies to deal with long sequence dependency problems, prevent the gradient disappearance problem and have good learning ability to build a time series analysis model, monitor network traffic and log records in real time, and identify abnormal behavior patterns; at the same time, use deep learning classifiers to accurately classify threat types.

7. The platform according to claim 4, It is characterized in that Dynamic threat warning and emergency response system, including Real-time early warning: Establish a machine learning-based early warning model to predict and identify potential risks or problems. Use mathematical, statistical and machine learning methods to analyze and model to discover potential risks or problems and issue early warnings. Emergency response recommendations: Combined with historical case libraries and knowledge bases, it contains expertise, experience, best practices, case studies, and solutions in the security field or industry, so that this knowledge can be accessed and utilized to support decision-making, solve problems, or improve work efficiency, generate targeted emergency response recommendations and mitigation measures, and assist personnel to take quick action.

8. The platform according to claim 7, It is characterized in that Immediately assess the urgency and potential impact of identified threats, and provide real-time notifications via API, email, and SMS.

Citation Information

Patent Citations

  • Text data-oriented threat intelligence knowledge graph construction method

    CN110717049A

  • Threat intelligence-oriented entity identification and relation extraction method

    CN113901818A

  • Secure data analysis method and system

    CN116662989A

  • Network information security protection system

    CN118353702A

Cited By

  • Intelligent prediction method for privacy information storage and transmission security situation

    CN119377615A

  • Network security monitoring and early warning method and device for service system, electronic equipment and storage medium

    CN120498861A

  • Supply chain security information push early warning method and device

    CN121037440A